00010000:00010000:0.0:1787827017.452379:0:92691:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e20000/0x531a24091f988510 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb26b3 expref: -99 pid: 92691 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827017.452390:0:92691:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827017.452399:0:92691:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=0 00000080:00200000:0.0:1787827017.452405:0:92691:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827017.452413:0:92691:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827017.452421:0:92691:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e20000/0x531a24091f988510 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb26b3 expref: -99 pid: 92691 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827017.452430:0:92691:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e20000/0x531a24091f988510 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb26b3 expref: -99 pid: 92691 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827017.452437:0:92691:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827017.452441:0:92691:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827017.452444:0:92691:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988510 00000080:00010000:0.0:1787827017.452447:0:92691:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988510 from it ffffbe19085fbd68 00010000:00010000:0.0:1787827017.452449:0:92691:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e20000/0x531a24091f988510 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb26b3 expref: -99 pid: 92691 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827017.452457:0:92691:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e20000/0x531a24091f988510 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb26b3 expref: -99 pid: 92691 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827017.452464:0:92691:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:0.0:1787827017.452470:0:92691:0:(namei.c:2418:ll_unlink()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248) 00010000:00010000:0.0:1787827017.452500:0:92691:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e20000/0x531a24091f988510 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb26b3 expref: -99 pid: 92691 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827017.452525:0:92691:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012e00/0x531a24091f9884fb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1e7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bb2682 expref: -99 pid: 92652 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827017.452541:0:92691:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x1e7:0x0]: losing layout lock 00010000:00010000:0.0:1787827017.452577:0:92691:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012e00/0x531a24091f9884fb lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1e7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb2682 expref: -99 pid: 92652 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827017.452586:0:92691:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e20000/0x531a24091f988510 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb26b3 expref: -99 pid: 92691 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827017.452604:0:92691:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787827017.452607:0:92691:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012e00/0x531a24091f9884fb lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1e7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb2682 expref: -99 pid: 92652 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827017.452624:0:92691:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e20000/0x531a24091f988510 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb26b3 expref: -99 pid: 92691 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787827017.452644:0:92691:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787827017.452652:0:92691:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a788621c0 x1874668972681600/t0(0) o36->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827017.452674:0:92691:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:92691:1874668972681600:192.168.203.148@tcp:36:rm.0 00000100:00100000:0.0:1787827017.452743:0:92691:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43695480 going to sleep for 12 seconds 00000100:00100000:0.0:1787827017.454561:0:92691:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:92691:1874668972681600:192.168.203.148@tcp:36:rm.0 00000080:00200000:0.0:1787827017.454622:0:92691:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x1e7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827017.454632:0:92691:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1e7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827017.454639:0:92691:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x1e7:0x0](ffff9c1a6cc1f448) 00010000:00010000:0.0:1787827019.191615:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827022.519667:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2411:1874668972682240:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827022.520079:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972682496:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827022.520133:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972682752:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827022.521721:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2411:1874668972682240:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827022.521733:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972682496:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827022.521740:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972682752:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827024.051756:0:92858:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:92858:1874668972684032:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827024.051774:0:92858:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827024.051785:0:92858:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827024.051796:0:92858:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827024.051805:0:92858:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827024.051814:0:92858:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827024.051822:0:92858:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827024.051833:0:92858:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827024.051843:0:92858:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a49d8d680 x1874668972684032/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827041 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827024.051862:0:92858:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827024.051873:0:92858:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827024.051877:0:92858:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827024.051905:0:92858:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827024.051911:0:92858:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98853a 00000080:00010000:0.0:1787827024.051915:0:92858:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827024.051919:0:92858:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98853a from it ffffbe1908773bf8 00010000:00010000:0.0:1787827024.051922:0:92858:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827024.051934:0:92858:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827024.051957:0:92858:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827024.051973:0:92858:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827024.051983:0:92858:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827024.051991:0:92858:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827024.052011:0:92858:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827024.052017:0:92858:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827024.052021:0:92858:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827024.052025:0:92858:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98853a 00000080:00010000:0.0:1787827024.052028:0:92858:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98853a from it ffffbe1908773b38 00010000:00010000:0.0:1787827024.052031:0:92858:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827024.052037:0:92858:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827024.052044:0:92858:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:0.0:1787827024.052048:0:92858:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), file ffff9c1a49677380, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:0.0:1787827024.052057:0:92858:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000002:00010000:0.0:1787827024.052073:0:92858:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000402:0x1e9:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:0.0:1787827024.052083:0:92858:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827024.052118:0:92858:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827024.052124:0:92858:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827024.052127:0:92858:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f98853a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2762 expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827024.052158:0:92858:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 92858 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827024.052164:0:92858:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 92858 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827024.052173:0:92858:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827024.052179:0:92858:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78861a40 x1874668972684288/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827024.052188:0:92858:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 92858 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827024.052210:0:92858:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:92858:1874668972684288:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827024.052311:0:92858:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43695780 going to sleep for 12 seconds 00010000:00010000:0.0:1787827024.311505:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827026.359549:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827029.338633:0:92858:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9c1a46db5000: committing for last_committed 8589944684 gen 1 00000100:00100000:0.0:1787827029.338662:0:92858:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9c1a78861a40 x1874668972684288/t8589944685(8589944685) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827041 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787827029.338675:0:92858:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:92858:1874668972684288:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827029.338696:0:92858:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827029.338733:0:92858:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1e9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827029.338755:0:92858:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827029.338765:0:92858:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827029.338774:0:92858:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827029.338784:0:92858:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827029.338792:0:92858:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827029.338798:0:92858:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827029.338807:0:92858:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9c1a78861a40 x1874668972684288/t8589944685(8589944685) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827041 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827029.338819:0:92858:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78861a40 x1874668972684288/t8589944685(8589944685) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827041 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827029.338829:0:92858:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827029.338845:0:92858:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589935618/489 (0 0) 00000080:00200000:0.0:1787827029.338862:0:92858:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827029.338875:0:92858:0:(namei.c:1628:ll_create_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000080:00200000:0.0:1787827029.338897:0:92858:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x1e9:0x0](ffff9c1a6cc1f448) 00000080:00200000:0.0:1787827029.338903:0:92858:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x1e9:0x0], updating i_size 0 00000080:00200000:0.0:1787827029.338935:0:92858:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x1e9:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1787827029.338941:0:92858:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x1e9:0x0](ffff9c1a6cc1f448) 00000080:00010000:0.0:1787827029.338945:0:92858:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1e9:0x0](ffff9c1a6cc1f448) for lock 0x531a24091f988541 00000080:00010000:0.0:1787827029.338953:0:92858:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1e9:0x0](ffff9c1a6cc1f448) 00000080:00200008:0.0:1787827029.338975:0:92858:0:(file.c:1090:ll_file_open()) START file f41i.sanityn:[0x200000402:0x1e9:0x0](ffff9c1a6cc1f448), flags 100300 00000002:00100000:0.0:1787827029.339000:0:92858:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78861a40 x1874668972684288/t8589944685(8589944685) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827041 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827029.339049:0:92858:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988541 from it ffff9c1a7a8ed720 00010000:00010000:0.0:1787827029.339056:0:92858:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827029.339071:0:92858:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988541 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1e9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb278c expref: -99 pid: 92858 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827029.339098:0:92858:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1e9:0x0](ffff9c1a6cc1f448), inode mode 81a4 mask 40 00000080:00200008:0.0:1787827029.339129:0:92858:0:(file.c:479:ll_file_release()) START file f41i.sanityn:[0x200000402:0x1e9:0x0](ffff9c1a6cc1f448), flags 100000 00000002:00100000:0.0:1787827029.339167:0:92858:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9c1a78861a40 x1874668972684288/t8589944685(8589944685) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827041 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827029.339178:0:92858:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827029.339186:0:92858:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78863480 x1874668972685696/t0(0) o35->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827029.339214:0:92858:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:92858:1874668972685696:192.168.203.148@tcp:35:multiop.0 00000100:00100000:0.0:1787827029.339326:0:92858:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43695780 going to sleep for 11 seconds 00000100:00100000:0.0:1787827029.340587:0:92858:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:92858:1874668972685696:192.168.203.148@tcp:35:multiop.0 00000002:00100000:0.0:1787827029.340606:0:92858:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9c1a78861a40 x1874668972684288/t8589944685(8589944685) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827041 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787827031.479540:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827033.527529:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827035.484031:0:92961:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9c1a46db5000: committing for last_committed 8589944687 gen 1 00000100:00100000:0.0:1787827035.484072:0:92961:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9c1a49d8e940 x1874668972686592/t8589944688(8589944688) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827047 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787827035.484113:0:92961:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:92961:1874668972686592:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827035.484137:0:92961:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827035.484187:0:92961:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1ea:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827035.484229:0:92961:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827035.484249:0:92961:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827035.484266:0:92961:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827035.484286:0:92961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827035.484297:0:92961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827035.484310:0:92961:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827035.484330:0:92961:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9c1a49d8e940 x1874668972686592/t8589944688(8589944688) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827047 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827035.484400:0:92961:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a49d8e940 x1874668972686592/t8589944688(8589944688) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827047 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827035.484413:0:92961:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827035.484435:0:92961:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589935618/490 (0 0) 00000080:00200000:0.0:1787827035.484444:0:92961:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827035.484470:0:92961:0:(namei.c:1628:ll_create_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000080:00200000:0.0:1787827035.485542:0:92961:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x1ea:0x0](ffff9c1a6cc1d348) 00000080:00200000:0.0:1787827035.485548:0:92961:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x1ea:0x0], updating i_size 0 00000080:00200000:0.0:1787827035.485578:0:92961:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x1ea:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1787827035.485583:0:92961:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x1ea:0x0](ffff9c1a6cc1d348) 00000080:00010000:0.0:1787827035.485586:0:92961:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1ea:0x0](ffff9c1a6cc1d348) for lock 0x531a24091f988564 00000080:00010000:0.0:1787827035.485596:0:92961:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1ea:0x0](ffff9c1a6cc1d348) 00000080:00200008:0.0:1787827035.485608:0:92961:0:(file.c:1090:ll_file_open()) START file f41i.sanityn:[0x200000402:0x1ea:0x0](ffff9c1a6cc1d348), flags 100300 00000002:00100000:0.0:1787827035.485628:0:92961:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a49d8e940 x1874668972686592/t8589944688(8589944688) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827047 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827035.485667:0:92961:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988564 from it ffff9c1a46e79900 00010000:00010000:0.0:1787827035.485671:0:92961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827035.485681:0:92961:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941a00/0x531a24091f988564 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1ea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb2811 expref: -99 pid: 92961 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827035.485699:0:92961:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1ea:0x0](ffff9c1a6cc1d348), inode mode 81a4 mask 40 00000080:00200008:0.0:1787827035.485748:0:92961:0:(file.c:479:ll_file_release()) START file f41i.sanityn:[0x200000402:0x1ea:0x0](ffff9c1a6cc1d348), flags 100000 00000002:00100000:0.0:1787827035.485779:0:92961:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9c1a49d8e940 x1874668972686592/t8589944688(8589944688) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827047 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827035.485788:0:92961:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827035.485792:0:92961:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78863480 x1874668972688128/t0(0) o35->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827035.485804:0:92961:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:92961:1874668972688128:192.168.203.148@tcp:35:multiop.0 00000100:00100000:0.0:1787827035.485847:0:92961:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a48163d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787827035.487562:0:92961:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:92961:1874668972688128:192.168.203.148@tcp:35:multiop.0 00000002:00100000:0.0:1787827035.487583:0:92961:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9c1a49d8e940 x1874668972686592/t8589944688(8589944688) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827047 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787827035.575514:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787827036.347043:0:93064:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827036.347070:0:93064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827036.347120:0:93064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 93064 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.347135:0:93064:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 93064 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.347146:0:93064:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 93064 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827036.347170:0:93064:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:93064:1874668972688640:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827036.347300:0:93064:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a48163d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787827036.348987:0:93064:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:93064:1874668972688640:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827036.349005:0:93064:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.349015:0:93064:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.349025:0:93064:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.349032:0:93064:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.349040:0:93064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.349045:0:93064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.349053:0:93064:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827036.349060:0:93064:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78861a40 x1874668972688640/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827053 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827036.349068:0:93064:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.349077:0:93064:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827036.349081:0:93064:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827036.349101:0:93064:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827036.349107:0:93064:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988580 00000080:00010000:0.0:1787827036.349111:0:93064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827036.349114:0:93064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988580 from it ffffbe19088cbbf8 00010000:00010000:0.0:1787827036.349118:0:93064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.349126:0:93064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827036.349145:0:93064:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827036.349162:0:93064:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827036.349170:0:93064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827036.349183:0:93064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827036.349197:0:93064:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827036.349204:0:93064:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827036.349208:0:93064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827036.349211:0:93064:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988580 00000080:00010000:0.0:1787827036.349215:0:93064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988580 from it ffffbe19088cbb38 00010000:00010000:0.0:1787827036.349218:0:93064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827036.349226:0:93064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827036.349234:0:93064:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:0.0:1787827036.349238:0:93064:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), file ffff9c1a78af5980, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:0.0:1787827036.349253:0:93064:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000002:00010000:0.0:1787827036.349270:0:93064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000402:0x1eb:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:0.0:1787827036.349280:0:93064:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827036.349312:0:93064:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827036.349317:0:93064:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827036.349319:0:93064:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988580 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb286c expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827036.349370:0:93064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 93064 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827036.349375:0:93064:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 93064 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827036.349382:0:93064:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827036.349387:0:93064:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78861a40 x1874668972688896/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827036.349393:0:93064:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 93064 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827036.349403:0:93064:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:93064:1874668972688896:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827036.349503:0:93064:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a48163d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787827037.879634:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972689152:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827037.879995:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972689408:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827037.880039:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972689664:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827037.880074:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972689920:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827037.881881:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972689152:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827037.881910:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972689408:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827037.881944:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972689664:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827037.881959:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972689920:192.168.203.148@tcp:400:kworker.0 00010000:00010000:0.0:1787827038.647561:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827040.695515:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827041.626411:0:93064:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:93064:1874668972688896:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827041.626454:0:93064:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827041.626494:0:93064:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1eb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827041.626504:0:93064:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827041.626518:0:93064:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827041.626527:0:93064:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827041.626549:0:93064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827041.626563:0:93064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827041.626581:0:93064:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827041.626609:0:93064:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9c1a78861a40 x1874668972688896/t8589944691(8589944691) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827053 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1787827041.626627:0:93064:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78861a40 x1874668972688896/t8589944691(8589944691) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827053 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827041.626644:0:93064:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827041.626655:0:93064:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589935618/491 (0 0) 00000080:00200000:0.0:1787827041.626660:0:93064:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1787827041.626671:0:93064:0:(namei.c:1628:ll_create_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000080:00200000:0.0:1787827041.628021:0:93064:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x1eb:0x0](ffff9c1a6cc1aa08) 00000080:00200000:0.0:1787827041.628028:0:93064:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x1eb:0x0], updating i_size 0 00000080:00200000:0.0:1787827041.628063:0:93064:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x1eb:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1787827041.628069:0:93064:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x1eb:0x0](ffff9c1a6cc1aa08) 00000080:00010000:0.0:1787827041.628071:0:93064:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1eb:0x0](ffff9c1a6cc1aa08) for lock 0x531a24091f988587 00000080:00010000:0.0:1787827041.628081:0:93064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1eb:0x0](ffff9c1a6cc1aa08) 00000080:00200008:0.0:1787827041.628093:0:93064:0:(file.c:1090:ll_file_open()) START file f41i.sanityn:[0x200000402:0x1eb:0x0](ffff9c1a6cc1aa08), flags 100300 00000002:00100000:0.0:1787827041.628115:0:93064:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78861a40 x1874668972688896/t8589944691(8589944691) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827053 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1787827041.628149:0:93064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988587 from it ffff9c1a7a8ed600 00010000:00010000:0.0:1787827041.628155:0:93064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827041.628164:0:93064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f988587 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1eb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb2896 expref: -99 pid: 93064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827041.628182:0:93064:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1eb:0x0](ffff9c1a6cc1aa08), inode mode 81a4 mask 40 00000080:00200008:0.0:1787827041.628207:0:93064:0:(file.c:479:ll_file_release()) START file f41i.sanityn:[0x200000402:0x1eb:0x0](ffff9c1a6cc1aa08), flags 100000 00000002:00100000:0.0:1787827041.628241:0:93064:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9c1a78861a40 x1874668972688896/t8589944691(8589944691) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827053 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1787827041.628250:0:93064:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1787827041.628254:0:93064:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78863c00 x1874668972690432/t0(0) o35->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827041.628270:0:93064:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:93064:1874668972690432:192.168.203.148@tcp:35:multiop.0 00000100:00100000:0.0:1787827041.628326:0:93064:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a48163d80 going to sleep for 11 seconds 00000100:00100000:0.0:1787827041.629909:0:93064:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9c1a46db5000: committing for last_committed 8589944690 gen 1 00000100:00100000:0.0:1787827041.629932:0:93064:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9c1a78861a40 x1874668972688896/t8589944691(8589944691) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827053 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787827041.629949:0:93064:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:93064:1874668972690432:192.168.203.148@tcp:35:multiop.0 00000002:00100000:0.0:1787827041.629966:0:93064:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9c1a78861a40 x1874668972688896/t8589944691(8589944691) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827053 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827042.663026:0:93169:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:0.0:1787827042.663052:0:93169:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827042.663065:0:93169:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787827042.663109:0:93169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827042.663119:0:93169:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827042.663132:0:93169:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827042.663155:0:93169:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:93169:1874668972691328:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827042.663244:0:93169:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a48163d80 going to sleep for 12 seconds 00010000:00010000:0.0:1787827042.743525:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827042.999578:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972691584:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827042.999686:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972691840:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827042.999735:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972692096:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827043.001649:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972691584:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827043.001662:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972691840:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827043.002000:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972692096:192.168.203.148@tcp:400:kworker.0 00010000:00010000:0.0:1787827045.815542:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827047.771370:0:93169:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:93169:1874668972691328:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827047.771406:0:93169:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.771418:0:93169:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.771430:0:93169:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.771445:0:93169:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.771456:0:93169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.771461:0:93169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.771466:0:93169:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827047.771474:0:93169:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78863480 x1874668972691328/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827059 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827047.771484:0:93169:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.771505:0:93169:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827047.771512:0:93169:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827047.771542:0:93169:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827047.771554:0:93169:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f9885b1 00000080:00010000:0.0:1787827047.771575:0:93169:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:0.0:1787827047.771581:0:93169:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9885b1 from it ffffbe190897bbf8 00010000:00010000:0.0:1787827047.771586:0:93169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.771600:0:93169:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827047.771628:0:93169:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 1 00000080:00200000:0.0:1787827047.771656:0:93169:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:0.0:1787827047.771670:0:93169:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827047.771680:0:93169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827047.771699:0:93169:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827047.771708:0:93169:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827047.771714:0:93169:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:0.0:1787827047.771717:0:93169:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f9885b1 00000080:00010000:0.0:1787827047.771722:0:93169:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9885b1 from it ffffbe190897bb38 00010000:00010000:0.0:1787827047.771726:0:93169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827047.771754:0:93169:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827047.771789:0:93169:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 3 00000080:00200000:0.0:1787827047.771795:0:93169:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), file ffff9c1a496eaa80, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:0.0:1787827047.771804:0:93169:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), intent=open|creat 00000002:00010000:0.0:1787827047.771824:0:93169:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000403:0x1bf1:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:0.0:1787827047.771834:0:93169:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827047.771893:0:93169:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827047.771901:0:93169:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827047.771904:0:93169:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013e00/0x531a24091f9885b1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2914 expref: -99 pid: 93169 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827047.771933:0:93169:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.771943:0:93169:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827047.771953:0:93169:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a49bbf800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827047.771961:0:93169:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78863480 x1874668972692608/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827047.771971:0:93169:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827047.771991:0:93169:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:93169:1874668972692608:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827047.772123:0:93169:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a48163d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787827047.775530:0:93169:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:93169:1874668972692608:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827047.775558:0:93169:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.775572:0:93169:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.775579:0:93169:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.775585:0:93169:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.775593:0:93169:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.775599:0:93169:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9c1a59013c00 already unused, calling callback (ffffffffc1bbd990) 00010000:00010000:0.0:1787827047.775604:0:93169:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.775611:0:93169:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.775618:0:93169:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 5/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.775628:0:93169:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827047.775638:0:93169:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827047.775644:0:93169:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013c00/0x531a24091f9885b8 lrc: 0/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 93169 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827047.775663:0:93169:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0xb, status=-17 req@ffff9c1a78863480 x1874668972692608/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827064 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827047.775678:0:93169:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -17 00010000:00010000:0.0:1787827047.864428:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827048.119617:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972692864:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827048.119703:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972693120:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827048.121607:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972692864:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827048.121629:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972693120:192.168.203.148@tcp:400:kworker.0 00000080:00200000:0.0:1787827048.214941:0:93206:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827048.214962:0:93206:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827048.215014:0:93206:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 93206 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827048.215024:0:93206:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 93206 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827048.215034:0:93206:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 93206 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827048.215057:0:93206:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:93206:1874668972693376:192.168.203.148@tcp:101:rm.0 00000100:00100000:0.0:1787827048.215138:0:93206:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a48163d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787827048.216685:0:93206:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:93206:1874668972693376:192.168.203.148@tcp:101:rm.0 00010000:00010000:0.0:1787827048.216705:0:93206:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827048.216716:0:93206:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827048.216729:0:93206:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827048.216752:0:93206:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827048.216766:0:93206:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827048.216774:0:93206:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827048.216781:0:93206:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827048.216791:0:93206:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a788612c0 x1874668972693376/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827065 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827048.216804:0:93206:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827048.216815:0:93206:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827048.216821:0:93206:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827048.216894:0:93206:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827048.216900:0:93206:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9885bf 00000080:00010000:0.0:1787827048.216906:0:93206:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827048.216909:0:93206:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9885bf from it ffffbe19089cbb58 00010000:00010000:0.0:1787827048.216912:0:93206:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827048.216926:0:93206:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827048.216945:0:93206:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827048.216953:0:93206:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=8 00000080:00200008:0.0:1787827048.216962:0:93206:0:(file.c:6313:ll_getattr_dentry()) START file f41i.sanityn:[0x200000402:0x1ec:0x0](ffff9c1a813f80c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827048.216966:0:93206:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1ec:0x0](ffff9c1a813f80c8),name=f41i.sanityn 00000002:00010000:0.0:1787827048.216981:0:93206:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1ec:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827048.216990:0:93206:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7200/0x531a24091f9885aa lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb291b expref: -99 pid: 93167 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827048.216999:0:93206:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7200/0x531a24091f9885aa lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb291b expref: -99 pid: 93167 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827048.217006:0:93206:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827048.217019:0:93206:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1ec:0x0](ffff9c1a813f80c8) 00000080:00010000:0.0:1787827048.217022:0:93206:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1ec:0x0](ffff9c1a813f80c8) for lock 0x531a24091f9885aa 00000080:00010000:0.0:1787827048.217025:0:93206:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9885aa from it ffffbe19089cbcb8 00010000:00010000:0.0:1787827048.217028:0:93206:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7200/0x531a24091f9885aa lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb291b expref: -99 pid: 93167 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827048.217037:0:93206:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7200/0x531a24091f9885aa lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb291b expref: -99 pid: 93167 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827048.217069:0:93206:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827048.217076:0:93206:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827048.217082:0:93206:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827048.217090:0:93206:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827048.217105:0:93206:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827048.217108:0:93206:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827048.217111:0:93206:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9885bf 00000080:00010000:0.0:1787827048.217114:0:93206:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9885bf from it ffffbe19089cbc10 00010000:00010000:0.0:1787827048.217117:0:93206:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827048.217124:0:93206:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827048.217132:0:93206:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827048.217137:0:93206:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=0 00000080:00200000:0.0:1787827048.217141:0:93206:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827048.217146:0:93206:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827048.217151:0:93206:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827048.217159:0:93206:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827048.217165:0:93206:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827048.217168:0:93206:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827048.217170:0:93206:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9885bf 00000080:00010000:0.0:1787827048.217174:0:93206:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9885bf from it ffffbe19089cbd68 00010000:00010000:0.0:1787827048.217177:0:93206:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827048.217184:0:93206:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827048.217191:0:93206:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:0.0:1787827048.217195:0:93206:0:(namei.c:2418:ll_unlink()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248) 00010000:00010000:0.0:1787827048.217215:0:93206:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827048.217235:0:93206:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7200/0x531a24091f9885aa lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1ec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bb291b expref: -99 pid: 93167 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827048.217244:0:93206:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x1ec:0x0]: losing layout lock 00010000:00010000:0.0:1787827048.217266:0:93206:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7200/0x531a24091f9885aa lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb291b expref: -99 pid: 93167 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827048.217273:0:93206:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827048.217278:0:93206:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787827048.217281:0:93206:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7200/0x531a24091f9885aa lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1ec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb291b expref: -99 pid: 93167 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827048.217295:0:93206:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f9885bf lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb294c expref: -99 pid: 93206 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787827048.217316:0:93206:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787827048.217322:0:93206:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a788612c0 x1874668972693632/t0(0) o36->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827048.217338:0:93206:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:93206:1874668972693632:192.168.203.148@tcp:36:rm.0 00000100:00100000:0.0:1787827048.217619:0:93206:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a48163d80 going to sleep for 12 seconds 00000100:00100000:0.0:1787827048.219861:0:93206:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:93206:1874668972693632:192.168.203.148@tcp:36:rm.0 00000080:00200000:0.0:1787827048.219904:0:93206:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x1ec:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827048.219913:0:93206:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1ec:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827048.219920:0:93206:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x1ec:0x0](ffff9c1a813f80c8) 00010000:00010000:0.0:1787827049.911543:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827052.983601:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827053.239690:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972694272:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827053.239838:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972694528:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827053.239925:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972694784:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827053.239975:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972695040:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827053.241501:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972694272:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827053.242019:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972695040:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827053.242032:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972694528:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827053.242045:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972694784:192.168.203.148@tcp:400:kworker.0 00010000:00010000:0.0:1787827055.031507:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827056.055519:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827057.079498:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827058.359620:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972696704:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827058.359703:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972696960:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827058.359737:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972697216:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827058.361880:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972696704:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827058.361904:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972696960:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827058.361924:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972697216:192.168.203.148@tcp:400:kworker.0 00010000:00010000:0.0:1787827060.152448:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827062.199988:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827063.223529:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827064.247604:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827067.319514:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827069.369399:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827070.391547:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827071.415521:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827074.487558:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827076.535499:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827077.559570:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827078.584455:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827078.839584:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2411:1874668972704896:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827078.839669:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972705152:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827078.839703:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972705408:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827078.839747:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972705664:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827078.841504:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2411:1874668972704896:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827078.841521:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972705152:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827078.841536:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972705408:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827078.841551:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972705664:192.168.203.148@tcp:400:kworker.0 00010000:00010000:0.0:1787827081.655534:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827083.703578:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827083.959700:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972706816:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827083.959830:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972707072:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827083.959874:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972707328:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827083.961680:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972706816:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827083.961709:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972707072:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827083.961727:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972707328:192.168.203.148@tcp:400:kworker.0 00010000:00010000:0.0:1787827084.727453:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827085.751539:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827088.823524:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827090.871455:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787827091.179764:0:93927:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827091.179786:0:93927:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827091.179834:0:93927:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 93927 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827091.179841:0:93927:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 93927 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827091.179848:0:93927:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 93927 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827091.179866:0:93927:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:93927:1874668972710400:192.168.203.148@tcp:101:rm.0 00000100:00100000:0.0:1787827091.179957:0:93927:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a48163c00 going to sleep for 12 seconds 00000100:00100000:0.0:1787827091.181323:0:93927:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:93927:1874668972710400:192.168.203.148@tcp:101:rm.0 00010000:00010000:0.0:1787827091.181336:0:93927:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827091.181360:0:93927:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827091.181370:0:93927:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827091.181381:0:93927:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827091.181392:0:93927:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827091.181400:0:93927:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827091.181409:0:93927:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827091.181421:0:93927:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78863840 x1874668972710400/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827108 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827091.181430:0:93927:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827091.181439:0:93927:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827091.181442:0:93927:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827091.181460:0:93927:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827091.181464:0:93927:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9886b4 00000080:00010000:0.0:1787827091.181467:0:93927:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827091.181470:0:93927:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9886b4 from it ffffbe1908fbbb58 00010000:00010000:0.0:1787827091.181472:0:93927:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827091.181477:0:93927:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827091.181491:0:93927:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827091.181498:0:93927:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=8 00000080:00200008:0.0:1787827091.181504:0:93927:0:(file.c:6313:ll_getattr_dentry()) START file f41i.sanityn:[0x200000402:0x1f3:0x0](ffff9c1a813fec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827091.181507:0:93927:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1f3:0x0](ffff9c1a813fec08),name=f41i.sanityn 00000002:00010000:0.0:1787827091.181517:0:93927:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1f3:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827091.181524:0:93927:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98869f lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb2cbe expref: -99 pid: 93888 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827091.181530:0:93927:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98869f lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1f3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb2cbe expref: -99 pid: 93888 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827091.181535:0:93927:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827091.181537:0:93927:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1f3:0x0](ffff9c1a813fec08) 00000080:00010000:0.0:1787827091.181539:0:93927:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1f3:0x0](ffff9c1a813fec08) for lock 0x531a24091f98869f 00000080:00010000:0.0:1787827091.181541:0:93927:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98869f from it ffffbe1908fbbcb8 00010000:00010000:0.0:1787827091.181542:0:93927:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98869f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1f3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb2cbe expref: -99 pid: 93888 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827091.181546:0:93927:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98869f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1f3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb2cbe expref: -99 pid: 93888 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827091.181570:0:93927:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827091.181574:0:93927:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827091.181577:0:93927:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827091.181582:0:93927:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827091.181586:0:93927:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827091.181588:0:93927:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827091.181589:0:93927:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9886b4 00000080:00010000:0.0:1787827091.181591:0:93927:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9886b4 from it ffffbe1908fbbc10 00010000:00010000:0.0:1787827091.181592:0:93927:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827091.181597:0:93927:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827091.181601:0:93927:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827091.181605:0:93927:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=0 00000080:00200000:0.0:1787827091.181607:0:93927:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827091.181610:0:93927:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827091.181613:0:93927:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827091.181617:0:93927:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827091.181621:0:93927:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827091.181622:0:93927:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827091.181624:0:93927:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9886b4 00000080:00010000:0.0:1787827091.181625:0:93927:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9886b4 from it ffffbe1908fbbd68 00010000:00010000:0.0:1787827091.181627:0:93927:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827091.181631:0:93927:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827091.181638:0:93927:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:0.0:1787827091.181642:0:93927:0:(namei.c:2418:ll_unlink()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248) 00010000:00010000:0.0:1787827091.181664:0:93927:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827091.181689:0:93927:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98869f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1f3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bb2cbe expref: -99 pid: 93888 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827091.181699:0:93927:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x1f3:0x0]: losing layout lock 00010000:00010000:0.0:1787827091.181733:0:93927:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98869f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1f3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb2cbe expref: -99 pid: 93888 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827091.181741:0:93927:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827091.181748:0:93927:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787827091.181750:0:93927:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98869f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1f3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb2cbe expref: -99 pid: 93888 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827091.181764:0:93927:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013600/0x531a24091f9886b4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2cef expref: -99 pid: 93927 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787827091.181781:0:93927:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787827091.181788:0:93927:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78863840 x1874668972710656/t0(0) o36->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827091.181803:0:93927:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:93927:1874668972710656:192.168.203.148@tcp:36:rm.0 00000100:00100000:0.0:1787827091.181842:0:93927:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a48163c00 going to sleep for 12 seconds 00010000:00010000:0.0:1787827091.895489:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827092.919558:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827098.039562:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787827098.101489:0:94096:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:0.0:1787827098.101518:0:94096:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827098.101533:0:94096:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787827098.101581:0:94096:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827098.101591:0:94096:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827098.101601:0:94096:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827098.101625:0:94096:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94096:1874668972713472:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827098.101841:0:94096:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146600 going to sleep for 12 seconds 00010000:00010000:0.0:1787827099.063514:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827099.319622:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972713728:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827099.319710:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972713984:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827099.319752:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972714240:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827099.321572:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972713728:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827099.321585:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972713984:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827099.321956:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972714240:192.168.203.148@tcp:400:kworker.0 00010000:00010000:0.0:1787827100.087523:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827103.066681:0:94096:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94096:1874668972713472:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827103.066707:0:94096:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.066740:0:94096:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.066756:0:94096:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.066781:0:94096:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.066821:0:94096:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.066838:0:94096:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.066852:0:94096:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827103.066866:0:94096:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78863c00 x1874668972713472/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827115 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827103.066889:0:94096:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.066923:0:94096:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827103.066935:0:94096:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827103.066961:0:94096:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827103.066971:0:94096:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f9886ec 00000080:00010000:0.0:1787827103.066978:0:94096:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:0.0:1787827103.066981:0:94096:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9886ec from it ffffbe19091abbf8 00010000:00010000:0.0:1787827103.066985:0:94096:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.067012:0:94096:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827103.067043:0:94096:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 1 00000080:00200000:0.0:1787827103.067072:0:94096:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:0.0:1787827103.067090:0:94096:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827103.067114:0:94096:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827103.067140:0:94096:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827103.067152:0:94096:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827103.067159:0:94096:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:0.0:1787827103.067162:0:94096:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f9886ec 00000080:00010000:0.0:1787827103.067167:0:94096:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9886ec from it ffffbe19091abb38 00010000:00010000:0.0:1787827103.067172:0:94096:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827103.067189:0:94096:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827103.067205:0:94096:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 3 00000080:00200000:0.0:1787827103.067211:0:94096:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), file ffff9c1a463fd680, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:0.0:1787827103.067219:0:94096:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), intent=open|creat 00000002:00010000:0.0:1787827103.067239:0:94096:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000403:0x1bfa:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:0.0:1787827103.067255:0:94096:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827103.067313:0:94096:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827103.067322:0:94096:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827103.067326:0:94096:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9886ec lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2dc1 expref: -99 pid: 94096 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827103.067392:0:94096:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.067417:0:94096:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827103.067429:0:94096:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a49bbf800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827103.067438:0:94096:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78863c00 x1874668972714880/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827103.067448:0:94096:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827103.067461:0:94096:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94096:1874668972714880:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827103.067553:0:94096:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146600 going to sleep for 12 seconds 00000100:00100000:0.0:1787827103.069292:0:94096:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94096:1874668972714880:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827103.069306:0:94096:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.069316:0:94096:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.069323:0:94096:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.069330:0:94096:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.069338:0:94096:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.069375:0:94096:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9c1a59013a00 already unused, calling callback (ffffffffc1bbd990) 00010000:00010000:0.0:1787827103.069381:0:94096:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.069389:0:94096:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.069397:0:94096:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 5/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.069405:0:94096:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827103.069412:0:94096:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827103.069423:0:94096:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013a00/0x531a24091f9886f3 lrc: 0/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 94096 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827103.069437:0:94096:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0xb, status=-17 req@ffff9c1a78863c00 x1874668972714880/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827120 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827103.069451:0:94096:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -17 00000100:00100000:0.0:1787827104.439602:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972716032:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827104.439709:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972716288:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827104.441105:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972716032:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827104.441458:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972716288:192.168.203.148@tcp:400:kworker.0 00010000:00010000:0.0:1787827105.207539:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827106.231528:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827107.255863:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827112.375524:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827113.399621:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827114.423552:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827114.679669:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972718976:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827114.679751:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972719232:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827114.679795:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972719488:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827114.682154:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972718976:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827114.682182:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972719232:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827114.682207:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972719488:192.168.203.148@tcp:400:kworker.0 00000080:00200000:0.0:1787827116.328550:0:94403:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827116.328578:0:94403:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827116.328635:0:94403:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988747 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94403 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827116.328647:0:94403:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988747 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94403 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827116.328657:0:94403:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988747 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94403 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827116.328682:0:94403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94403:1874668972720640:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827116.328775:0:94403:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146f00 going to sleep for 12 seconds 00000080:00200000:0.0:1787827116.533529:0:94405:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:0.0:1787827116.533556:0:94405:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827116.533569:0:94405:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787827116.533616:0:94405:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827116.533626:0:94405:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827116.533634:0:94405:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827116.533654:0:94405:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94405:1874668972721024:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827116.533744:0:94405:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146f00 going to sleep for 12 seconds 00010000:00010000:0.0:1787827119.543694:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827120.567792:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827121.498148:0:94405:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94405:1874668972721024:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827121.498180:0:94405:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.498204:0:94405:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.498217:0:94405:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.498229:0:94405:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.498250:0:94405:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.498270:0:94405:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.498283:0:94405:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827121.498303:0:94405:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78860000 x1874668972721024/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827133 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827121.498331:0:94405:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.498455:0:94405:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827121.498463:0:94405:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827121.498482:0:94405:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827121.498490:0:94405:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f988755 00000080:00010000:0.0:1787827121.498496:0:94405:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:0.0:1787827121.498499:0:94405:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988755 from it ffffbe190945bbf8 00010000:00010000:0.0:1787827121.498502:0:94405:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.498512:0:94405:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827121.498542:0:94405:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 1 00000080:00200000:0.0:1787827121.498575:0:94405:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:0.0:1787827121.498592:0:94405:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827121.498602:0:94405:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827121.498616:0:94405:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827121.498630:0:94405:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827121.498637:0:94405:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:0.0:1787827121.498641:0:94405:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f988755 00000080:00010000:0.0:1787827121.498646:0:94405:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988755 from it ffffbe190945bb38 00010000:00010000:0.0:1787827121.498654:0:94405:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827121.498663:0:94405:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827121.498672:0:94405:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 3 00000080:00200000:0.0:1787827121.498678:0:94405:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), file ffff9c1a4816a180, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:0.0:1787827121.498686:0:94405:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), intent=open|creat 00000002:00010000:0.0:1787827121.498708:0:94405:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000403:0x1bfd:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:0.0:1787827121.498721:0:94405:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827121.498768:0:94405:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827121.498783:0:94405:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827121.498788:0:94405:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012e00/0x531a24091f988755 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2f50 expref: -99 pid: 94405 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827121.498814:0:94405:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.498822:0:94405:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827121.498831:0:94405:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a49bbf800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827121.498838:0:94405:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78860000 x1874668972722432/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827121.498848:0:94405:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827121.498878:0:94405:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94405:1874668972722432:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827121.499017:0:94405:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146f00 going to sleep for 12 seconds 00000100:00100000:0.0:1787827121.500788:0:94405:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94405:1874668972722432:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827121.500802:0:94405:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.500811:0:94405:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.500818:0:94405:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.500825:0:94405:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.500832:0:94405:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.500839:0:94405:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9c1a59012c00 already unused, calling callback (ffffffffc1bbd990) 00010000:00010000:0.0:1787827121.500844:0:94405:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.500850:0:94405:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.500857:0:94405:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 5/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.500880:0:94405:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827121.500888:0:94405:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827121.500895:0:94405:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f98875c lrc: 0/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 94405 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827121.500914:0:94405:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0xb, status=-17 req@ffff9c1a78860000 x1874668972722432/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827138 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827121.500926:0:94405:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -17 00010000:00010000:0.0:1787827121.591896:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1787827122.002740:0:94442:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x1f8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1787827122.002749:0:94442:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1f8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787827122.002756:0:94442:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x1f8:0x0](ffff9c1a813fec08) 00000080:00200000:1.0:1787827122.551059:0:94506:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827122.551084:0:94506:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827122.551130:0:94506:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94506 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.551137:0:94506:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94506 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.551144:0:94506:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94506 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827122.551165:0:94506:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94506:1874668972722944:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827122.551266:0:94506:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1a80 going to sleep for 12 seconds 00000100:00100000:1.0:1787827122.554831:0:94506:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94506:1874668972722944:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827122.554868:0:94506:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.554881:0:94506:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.554891:0:94506:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.554902:0:94506:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.554916:0:94506:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.554922:0:94506:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.554927:0:94506:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827122.554935:0:94506:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78a7a580 x1874668972722944/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827139 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827122.554952:0:94506:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.554964:0:94506:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827122.554971:0:94506:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827122.555001:0:94506:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827122.555011:0:94506:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98876a 00000080:00010000:1.0:1787827122.555016:0:94506:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827122.555019:0:94506:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98876a from it ffffbe1909583bf8 00010000:00010000:1.0:1787827122.555022:0:94506:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.555031:0:94506:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827122.555053:0:94506:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827122.555073:0:94506:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827122.555086:0:94506:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827122.555095:0:94506:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827122.555104:0:94506:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827122.555111:0:94506:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827122.555114:0:94506:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827122.555117:0:94506:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98876a 00000080:00010000:1.0:1787827122.555120:0:94506:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98876a from it ffffbe1909583b38 00010000:00010000:1.0:1787827122.555123:0:94506:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827122.555129:0:94506:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827122.555137:0:94506:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:1.0:1787827122.555142:0:94506:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), file ffff9c1a59125680, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:1.0:1787827122.555156:0:94506:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000002:00010000:1.0:1787827122.555172:0:94506:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000402:0x1f9:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:1.0:1787827122.555182:0:94506:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827122.555218:0:94506:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827122.555225:0:94506:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827122.555227:0:94506:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98876a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2fb2 expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827122.555263:0:94506:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94506 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827122.555268:0:94506:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94506 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787827122.555276:0:94506:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787827122.555280:0:94506:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a7a580 x1874668972723200/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787827122.555286:0:94506:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94506 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827122.555310:0:94506:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94506:1874668972723200:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827122.555476:0:94506:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1a80 going to sleep for 12 seconds 00010000:00010000:1.0:1787827122.615497:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827124.664028:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827125.687583:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827126.711552:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827127.643725:0:94508:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94508:1874668972723328:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827127.643725:0:94506:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94506:1874668972723200:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827127.643760:0:94508:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827127.643762:0:94506:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.643778:0:94508:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.643789:0:94508:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.643799:0:94508:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827127.643802:0:94506:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827127.643808:0:94506:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.643809:0:94508:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.643840:0:94508:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827127.643852:0:94506:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827127.643868:0:94506:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.643873:0:94508:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827127.643887:0:94506:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827127.643889:0:94508:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a7895f0c0 x1874668972723328/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827139 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1787827127.643908:0:94506:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827127.643912:0:94508:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827127.643922:0:94506:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.643927:0:94508:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827127.643937:0:94508:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000002:00100000:1.0:1787827127.643939:0:94506:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9c1a78a7a580 x1874668972723200/t8589944733(8589944733) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827139 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787827127.643954:0:94506:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78a7a580 x1874668972723200/t8589944733(8589944733) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827139 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827127.643966:0:94506:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827127.643968:0:94508:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827127.643976:0:94508:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f988778 00000080:00010000:0.0:1787827127.643981:0:94508:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00010000:00010000:1.0:1787827127.643981:0:94506:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589935618/505 (0 0) 00000080:00010000:0.0:1787827127.643987:0:94508:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988778 from it ffffbe190957bbf8 00010000:00010000:0.0:1787827127.643990:0:94508:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1787827127.643991:0:94506:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1787827127.644002:0:94508:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827127.644009:0:94506:0:(namei.c:1628:ll_create_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000080:00200000:0.0:1787827127.644025:0:94508:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 1 00000080:00200000:0.0:1787827127.644047:0:94508:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000080:00200000:1.0:1787827127.644052:0:94506:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x1f9:0x0](ffff9c1a813fec08) 00000080:00200000:1.0:1787827127.644058:0:94506:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x1f9:0x0], updating i_size 0 00000002:00010000:0.0:1787827127.644061:0:94508:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827127.644068:0:94508:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827127.644080:0:94508:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827127.644085:0:94508:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827127.644088:0:94508:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:0.0:1787827127.644090:0:94508:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f988778 00000080:00010000:0.0:1787827127.644093:0:94508:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988778 from it ffffbe190957bb38 00010000:00010000:0.0:1787827127.644095:0:94508:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827127.644099:0:94508:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827127.644105:0:94506:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x1f9:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00200000:0.0:1787827127.644109:0:94508:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 3 00000080:00010000:1.0:1787827127.644112:0:94506:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x1f9:0x0](ffff9c1a813fec08) 00000080:00200000:0.0:1787827127.644114:0:94508:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), file ffff9c1a4326c780, open_flags 80c0, mode 81b6 opened 0 00000080:00010000:1.0:1787827127.644118:0:94506:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1f9:0x0](ffff9c1a813fec08) for lock 0x531a24091f988771 00000080:00200000:0.0:1787827127.644124:0:94508:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), intent=open|creat 00000080:00010000:1.0:1787827127.644138:0:94506:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1f9:0x0](ffff9c1a813fec08) 00000002:00010000:0.0:1787827127.644149:0:94508:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000403:0x1bfe:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00000080:00200008:1.0:1787827127.644155:0:94506:0:(file.c:1090:ll_file_open()) START file f41i.sanityn:[0x200000402:0x1f9:0x0](ffff9c1a813fec08), flags 100300 00010000:00010000:0.0:1787827127.644162:0:94508:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:1.0:1787827127.644178:0:94506:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78a7a580 x1874668972723200/t8589944733(8589944733) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827139 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787827127.644228:0:94508:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1787827127.644235:0:94506:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988771 from it ffff9c1a52928900 00010000:00010000:0.0:1787827127.644237:0:94508:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827127.644245:0:94508:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e21c00/0x531a24091f988778 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb2fd5 expref: -99 pid: 94508 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827127.644245:0:94506:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827127.644268:0:94506:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827127.644291:0:94506:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1f9:0x0](ffff9c1a813fec08), inode mode 81a4 mask 40 00010000:00010000:0.0:1787827127.644304:0:94508:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.644312:0:94508:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827127.644324:0:94508:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a49bbf800: modify RPC slot 1 is allocated opc 101, max 7 00000080:00200008:1.0:1787827127.644329:0:94506:0:(file.c:479:ll_file_release()) START file f41i.sanityn:[0x200000402:0x1f9:0x0](ffff9c1a813fec08), flags 100000 00000100:00100000:0.0:1787827127.644332:0:94508:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78861e00 x1874668972724480/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827127.644344:0:94508:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827127.644446:0:94506:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9c1a78a7a580 x1874668972723200/t8589944733(8589944733) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827139 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787827127.644457:0:94506:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787827127.644462:0:94506:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a7bc00 x1874668972724736/t0(0) o35->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1787827127.644477:0:94508:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94508:1874668972724480:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827127.644482:0:94506:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94506:1874668972724736:192.168.203.148@tcp:35:multiop.0 00000100:00100000:1.0:1787827127.644578:0:94506:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1a80 going to sleep for 11 seconds 00000100:00100000:0.0:1787827127.644579:0:94508:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146f00 going to sleep for 12 seconds 00000100:00100000:1.0:1787827127.645753:0:94506:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9c1a46db5000: committing for last_committed 8589944732 gen 1 00000100:00100000:1.0:1787827127.645773:0:94506:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9c1a78a7a580 x1874668972723200/t8589944733(8589944733) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827139 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1787827127.645780:0:94506:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94506:1874668972724736:192.168.203.148@tcp:35:multiop.0 00000002:00100000:1.0:1787827127.645793:0:94506:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9c1a78a7a580 x1874668972723200/t8589944733(8589944733) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827139 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787827127.646233:0:94508:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94508:1874668972724480:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827127.646248:0:94508:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.646258:0:94508:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.646271:0:94508:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.646279:0:94508:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.646289:0:94508:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.646297:0:94508:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9c1a59012400 already unused, calling callback (ffffffffc1bbd990) 00010000:00010000:0.0:1787827127.646302:0:94508:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.646309:0:94508:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.646317:0:94508:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 5/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.646324:0:94508:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827127.646339:0:94508:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827127.646373:0:94508:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012400/0x531a24091f98877f lrc: 0/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 94508 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827127.646389:0:94508:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0xb, status=-17 req@ffff9c1a78861e00 x1874668972724480/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827144 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827127.646400:0:94508:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -17 00000080:00200000:1.0:1787827128.038261:0:94545:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827128.038287:0:94545:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827128.038343:0:94545:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94545 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.038380:0:94545:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94545 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.038390:0:94545:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94545 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827128.038414:0:94545:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94545:1874668972724864:192.168.203.148@tcp:101:rm.0 00000100:00100000:1.0:1787827128.038501:0:94545:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1a80 going to sleep for 12 seconds 00000100:00100000:1.0:1787827128.040015:0:94545:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94545:1874668972724864:192.168.203.148@tcp:101:rm.0 00010000:00010000:1.0:1787827128.040032:0:94545:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.040043:0:94545:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.040054:0:94545:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.040070:0:94545:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.040081:0:94545:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.040089:0:94545:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.040096:0:94545:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827128.040107:0:94545:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78a79a40 x1874668972724864/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827145 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827128.040127:0:94545:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.040138:0:94545:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827128.040143:0:94545:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827128.040165:0:94545:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827128.040172:0:94545:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988786 00000080:00010000:1.0:1787827128.040177:0:94545:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827128.040187:0:94545:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988786 from it ffffbe19095c3b58 00010000:00010000:1.0:1787827128.040190:0:94545:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.040208:0:94545:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827128.040227:0:94545:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827128.040236:0:94545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=8 00000080:00200008:1.0:1787827128.040244:0:94545:0:(file.c:6313:ll_getattr_dentry()) START file f41i.sanityn:[0x200000402:0x1f9:0x0](ffff9c1a813fec08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787827128.040250:0:94545:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1f9:0x0](ffff9c1a813fec08),name=f41i.sanityn 00000002:00010000:1.0:1787827128.040264:0:94545:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1f9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827128.040272:0:94545:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827128.040281:0:94545:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827128.040288:0:94545:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827128.040292:0:94545:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1f9:0x0](ffff9c1a813fec08) 00000080:00010000:1.0:1787827128.040294:0:94545:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1f9:0x0](ffff9c1a813fec08) for lock 0x531a24091f988771 00000080:00010000:1.0:1787827128.040298:0:94545:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988771 from it ffffbe19095c3cb8 00010000:00010000:1.0:1787827128.040301:0:94545:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827128.040309:0:94545:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827128.040342:0:94545:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827128.040659:0:94545:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827128.040668:0:94545:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827128.040677:0:94545:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827128.040683:0:94545:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827128.040687:0:94545:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827128.040690:0:94545:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988786 00000080:00010000:1.0:1787827128.040694:0:94545:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988786 from it ffffbe19095c3c10 00010000:00010000:1.0:1787827128.040697:0:94545:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827128.040706:0:94545:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827128.040716:0:94545:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827128.040724:0:94545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=0 00000080:00200000:1.0:1787827128.040728:0:94545:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827128.040735:0:94545:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827128.040741:0:94545:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827128.040750:0:94545:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827128.040756:0:94545:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827128.040760:0:94545:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827128.040763:0:94545:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988786 00000080:00010000:1.0:1787827128.040767:0:94545:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988786 from it ffffbe19095c3d68 00010000:00010000:1.0:1787827128.040771:0:94545:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827128.040780:0:94545:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827128.040788:0:94545:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:1.0:1787827128.040794:0:94545:0:(namei.c:2418:ll_unlink()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248) 00010000:00010000:1.0:1787827128.040831:0:94545:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827128.040864:0:94545:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827128.040875:0:94545:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x1f9:0x0]: losing layout lock 00010000:00010000:1.0:1787827128.040897:0:94545:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827128.040905:0:94545:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827128.040911:0:94545:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1787827128.040914:0:94545:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988771 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1f9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb2fdc expref: -99 pid: 94506 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827128.040928:0:94545:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f988786 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb300d expref: -99 pid: 94545 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1787827128.040948:0:94545:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827128.040956:0:94545:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a79a40 x1874668972725120/t0(0) o36->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827128.040972:0:94545:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94545:1874668972725120:192.168.203.148@tcp:36:rm.0 00000100:00100000:1.0:1787827128.041024:0:94545:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1a80 going to sleep for 12 seconds 00000100:00100000:1.0:1787827128.042977:0:94545:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94545:1874668972725120:192.168.203.148@tcp:36:rm.0 00000080:00200000:1.0:1787827128.043012:0:94545:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x1f9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1787827128.043020:0:94545:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1f9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787827128.043025:0:94545:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x1f9:0x0](ffff9c1a813fec08) 00000080:00200000:1.0:1787827128.507296:0:94609:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827128.507324:0:94609:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827128.507407:0:94609:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94609 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.507419:0:94609:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94609 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.507430:0:94609:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94609 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827128.507455:0:94609:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94609:1874668972725248:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827128.507545:0:94609:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1a80 going to sleep for 12 seconds 00000100:00100000:1.0:1787827128.509644:0:94609:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94609:1874668972725248:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827128.509667:0:94609:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.509683:0:94609:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.509696:0:94609:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.509707:0:94609:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.509718:0:94609:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.509727:0:94609:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.509734:0:94609:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827128.509751:0:94609:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78a7b480 x1874668972725248/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827145 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827128.509765:0:94609:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.509777:0:94609:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827128.509784:0:94609:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827128.509805:0:94609:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827128.509810:0:94609:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98878d 00000080:00010000:1.0:1787827128.509828:0:94609:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827128.509831:0:94609:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98878d from it ffffbe190950bbf8 00010000:00010000:1.0:1787827128.509836:0:94609:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.509844:0:94609:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827128.509873:0:94609:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827128.509891:0:94609:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827128.509910:0:94609:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827128.509918:0:94609:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827128.509927:0:94609:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827128.509933:0:94609:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827128.509937:0:94609:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827128.509939:0:94609:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98878d 00000080:00010000:1.0:1787827128.509943:0:94609:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98878d from it ffffbe190950bb38 00010000:00010000:1.0:1787827128.509946:0:94609:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827128.509953:0:94609:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827128.509965:0:94609:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:1.0:1787827128.509969:0:94609:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), file ffff9c1a45f34900, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:1.0:1787827128.509976:0:94609:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000002:00010000:1.0:1787827128.509993:0:94609:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000402:0x1fa:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:1.0:1787827128.510001:0:94609:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827128.510030:0:94609:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827128.510035:0:94609:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827128.510037:0:94609:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6e00/0x531a24091f98878d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb3037 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827128.510059:0:94609:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94609 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.510065:0:94609:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94609 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787827128.510074:0:94609:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787827128.510080:0:94609:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a7b480 x1874668972725504/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787827128.510089:0:94609:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94609 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827128.510105:0:94609:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94609:1874668972725504:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827128.510215:0:94609:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1a80 going to sleep for 12 seconds 00000080:00200000:1.0:1787827128.711902:0:94611:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:1.0:1787827128.711940:0:94611:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827128.711955:0:94611:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:1.0:1787827128.712010:0:94611:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6e00/0x531a24091f98879b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 94611 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.712018:0:94611:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6e00/0x531a24091f98879b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94611 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827128.712026:0:94611:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6e00/0x531a24091f98879b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94611 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827128.712051:0:94611:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94611:1874668972725632:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827128.712153:0:94611:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1240 going to sleep for 12 seconds 00010000:00010000:0.0:1787827128.759520:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827129.783482:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827131.831540:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827132.855779:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827133.786644:0:94609:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9c1a46db5000: committing for last_committed 8589944735 gen 1 00000100:00100000:1.0:1787827133.786682:0:94609:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9c1a78a7b480 x1874668972725504/t8589944736(8589944736) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827145 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1787827133.786709:0:94609:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94609:1874668972725504:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827133.786728:0:94609:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827133.786794:0:94609:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1fa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827133.786827:0:94609:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827133.786855:0:94609:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827133.786867:0:94609:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827133.786878:0:94609:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827133.786891:0:94609:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827133.786907:0:94609:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827133.786929:0:94609:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9c1a78a7b480 x1874668972725504/t8589944736(8589944736) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827145 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787827133.786943:0:94609:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78a7b480 x1874668972725504/t8589944736(8589944736) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827145 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827133.786955:0:94609:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827133.786965:0:94609:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589935618/506 (0 0) 00000080:00200000:1.0:1787827133.786971:0:94609:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827133.786982:0:94609:0:(namei.c:1628:ll_create_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000080:00200000:1.0:1787827133.787020:0:94609:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x1fa:0x0](ffff9c1a813fec08) 00000080:00200000:1.0:1787827133.787028:0:94609:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x1fa:0x0], updating i_size 0 00000080:00200000:1.0:1787827133.787067:0:94609:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x1fa:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1787827133.787073:0:94609:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x1fa:0x0](ffff9c1a813fec08) 00000080:00010000:1.0:1787827133.787077:0:94609:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1fa:0x0](ffff9c1a813fec08) for lock 0x531a24091f988794 00000080:00010000:1.0:1787827133.787098:0:94609:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1fa:0x0](ffff9c1a813fec08) 00000080:00200008:1.0:1787827133.787113:0:94609:0:(file.c:1090:ll_file_open()) START file f41i.sanityn:[0x200000402:0x1fa:0x0](ffff9c1a813fec08), flags 100300 00000002:00100000:1.0:1787827133.787134:0:94609:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78a7b480 x1874668972725504/t8589944736(8589944736) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827145 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787827133.787185:0:94609:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988794 from it ffff9c1a529280c0 00010000:00010000:1.0:1787827133.787192:0:94609:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827133.787206:0:94609:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f988794 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb3061 expref: -99 pid: 94609 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827133.787225:0:94609:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1fa:0x0](ffff9c1a813fec08), inode mode 81a4 mask 40 00000080:00200008:1.0:1787827133.787264:0:94609:0:(file.c:479:ll_file_release()) START file f41i.sanityn:[0x200000402:0x1fa:0x0](ffff9c1a813fec08), flags 100000 00000002:00100000:1.0:1787827133.787294:0:94609:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9c1a78a7b480 x1874668972725504/t8589944736(8589944736) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827145 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787827133.787303:0:94609:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787827133.787307:0:94609:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a79a40 x1874668972727040/t0(0) o35->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787827133.787322:0:94609:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94609:1874668972727040:192.168.203.148@tcp:35:multiop.0 00000100:00100000:1.0:1787827133.787419:0:94609:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1787827133.789231:0:94609:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94609:1874668972727040:192.168.203.148@tcp:35:multiop.0 00000002:00100000:1.0:1787827133.789256:0:94609:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9c1a78a7b480 x1874668972725504/t8589944736(8589944736) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827145 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787827133.879504:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1787827134.609870:0:94712:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827134.609893:0:94712:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827134.609945:0:94712:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94712 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.609957:0:94712:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94712 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.609967:0:94712:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94712 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827134.609990:0:94712:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94712:1874668972727552:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827134.610077:0:94712:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1c00 going to sleep for 12 seconds 00000100:00100000:1.0:1787827134.611741:0:94712:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94712:1874668972727552:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827134.611759:0:94712:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.611770:0:94712:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.611794:0:94712:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.611813:0:94712:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.611825:0:94712:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.611833:0:94712:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.611841:0:94712:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827134.611851:0:94712:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78a7a580 x1874668972727552/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827151 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827134.611870:0:94712:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.611880:0:94712:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827134.611883:0:94712:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827134.611902:0:94712:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827134.611905:0:94712:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9887b0 00000080:00010000:1.0:1787827134.611909:0:94712:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827134.611911:0:94712:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887b0 from it ffffbe19096ebbf8 00010000:00010000:1.0:1787827134.611913:0:94712:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.611922:0:94712:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827134.611937:0:94712:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827134.611950:0:94712:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827134.611958:0:94712:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827134.611963:0:94712:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827134.611969:0:94712:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827134.611973:0:94712:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827134.611975:0:94712:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827134.611977:0:94712:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9887b0 00000080:00010000:1.0:1787827134.611979:0:94712:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887b0 from it ffffbe19096ebb38 00010000:00010000:1.0:1787827134.611981:0:94712:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827134.611985:0:94712:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827134.611989:0:94712:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:1.0:1787827134.611992:0:94712:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), file ffff9c1a51329980, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:1.0:1787827134.611997:0:94712:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000002:00010000:1.0:1787827134.612011:0:94712:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000402:0x1fb:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:1.0:1787827134.612021:0:94712:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827134.612054:0:94712:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827134.612060:0:94712:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827134.612063:0:94712:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6c00/0x531a24091f9887b0 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb30bc expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827134.612091:0:94712:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94712 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827134.612098:0:94712:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94712 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787827134.612106:0:94712:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787827134.612120:0:94712:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a7a580 x1874668972727808/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787827134.612129:0:94712:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94712 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827134.612144:0:94712:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94712:1874668972727808:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827134.612237:0:94712:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1c00 going to sleep for 12 seconds 00000080:00200000:0.0:1787827134.815659:0:94714:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:0.0:1787827134.815684:0:94714:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827134.815696:0:94714:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787827134.815733:0:94714:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827134.815739:0:94714:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827134.815748:0:94714:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827134.815823:0:94714:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94714:1874668972727936:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827134.815905:0:94714:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146000 going to sleep for 12 seconds 00010000:00010000:0.0:1787827135.927560:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827136.951649:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827138.999611:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827139.930553:0:94712:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94712:1874668972727808:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827139.930579:0:94712:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827139.930613:0:94712:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827139.930624:0:94712:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827139.930634:0:94712:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827139.930643:0:94712:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827139.930653:0:94712:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827139.930658:0:94712:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827139.930663:0:94712:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827139.930674:0:94712:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9c1a78a7a580 x1874668972727808/t8589944739(8589944739) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827151 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787827139.930687:0:94712:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78a7a580 x1874668972727808/t8589944739(8589944739) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827151 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827139.930698:0:94712:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827139.930709:0:94712:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589935618/507 (0 0) 00000080:00200000:1.0:1787827139.930715:0:94712:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827139.930726:0:94712:0:(namei.c:1628:ll_create_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000080:00200000:1.0:1787827139.930760:0:94712:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x1fb:0x0](ffff9c1a813f8908) 00000080:00200000:1.0:1787827139.930765:0:94712:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x1fb:0x0], updating i_size 0 00000080:00200000:1.0:1787827139.930791:0:94712:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x1fb:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1787827139.930794:0:94712:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x1fb:0x0](ffff9c1a813f8908) 00000080:00010000:1.0:1787827139.930796:0:94712:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1fb:0x0](ffff9c1a813f8908) for lock 0x531a24091f9887b7 00000080:00010000:1.0:1787827139.930804:0:94712:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1fb:0x0](ffff9c1a813f8908) 00000080:00200008:1.0:1787827139.930814:0:94712:0:(file.c:1090:ll_file_open()) START file f41i.sanityn:[0x200000402:0x1fb:0x0](ffff9c1a813f8908), flags 100300 00000002:00100000:1.0:1787827139.930830:0:94712:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78a7a580 x1874668972727808/t8589944739(8589944739) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827151 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787827139.930867:0:94712:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887b7 from it ffff9c1a529283c0 00010000:00010000:1.0:1787827139.930872:0:94712:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827139.930881:0:94712:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827139.930903:0:94712:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1fb:0x0](ffff9c1a813f8908), inode mode 81a4 mask 40 00000080:00200008:1.0:1787827139.930930:0:94712:0:(file.c:479:ll_file_release()) START file f41i.sanityn:[0x200000402:0x1fb:0x0](ffff9c1a813f8908), flags 100000 00000002:00100000:1.0:1787827139.930967:0:94712:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9c1a78a7a580 x1874668972727808/t8589944739(8589944739) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827151 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787827139.930977:0:94712:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787827139.930983:0:94712:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a7ad00 x1874668972728832/t0(0) o35->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787827139.931002:0:94712:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7ad00 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94712:1874668972728832:192.168.203.148@tcp:35:multiop.0 00000100:00100000:1.0:1787827139.931096:0:94712:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1c00 going to sleep for 11 seconds 00000100:00100000:0.0:1787827139.931818:0:94714:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94714:1874668972727936:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827139.931833:0:94714:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.931842:0:94714:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.931852:0:94714:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.931861:0:94714:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.931871:0:94714:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.931878:0:94714:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.931884:0:94714:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827139.931899:0:94714:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78860780 x1874668972727936/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827151 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827139.931911:0:94714:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.931921:0:94714:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827139.931927:0:94714:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827139.931945:0:94714:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827139.931950:0:94714:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f9887be 00000080:00010000:0.0:1787827139.931953:0:94714:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:0.0:1787827139.931956:0:94714:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887be from it ffffbe1909723bf8 00010000:00010000:0.0:1787827139.931959:0:94714:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.931966:0:94714:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827139.931981:0:94714:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 1 00000080:00200000:0.0:1787827139.932007:0:94714:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:0.0:1787827139.932020:0:94714:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827139.932028:0:94714:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827139.932037:0:94714:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827139.932048:0:94714:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827139.932051:0:94714:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:0.0:1787827139.932053:0:94714:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f9887be 00000080:00010000:0.0:1787827139.932056:0:94714:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887be from it ffffbe1909723b38 00010000:00010000:0.0:1787827139.932059:0:94714:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827139.932065:0:94714:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827139.932073:0:94714:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 3 00000080:00200000:0.0:1787827139.932078:0:94714:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), file ffff9c1a46cf5e00, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:0.0:1787827139.932085:0:94714:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), intent=open|creat 00000002:00010000:0.0:1787827139.932102:0:94714:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000403:0x1c00:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:0.0:1787827139.932111:0:94714:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827139.932151:0:94714:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827139.932157:0:94714:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827139.932160:0:94714:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012c00/0x531a24091f9887be lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb30df expref: -99 pid: 94714 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827139.932180:0:94714:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.932191:0:94714:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827139.932198:0:94714:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a49bbf800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827139.932204:0:94714:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78860780 x1874668972729088/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827139.932212:0:94714:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827139.932225:0:94714:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94714:1874668972729088:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827139.932322:0:94714:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146000 going to sleep for 12 seconds 00000100:00100000:1.0:1787827139.933335:0:94712:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9c1a46db5000: committing for last_committed 8589944738 gen 1 00000100:00100000:1.0:1787827139.933383:0:94712:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9c1a78a7a580 x1874668972727808/t8589944739(8589944739) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827151 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1787827139.933396:0:94712:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7ad00 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94712:1874668972728832:192.168.203.148@tcp:35:multiop.0 00000002:00100000:1.0:1787827139.933415:0:94712:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9c1a78a7a580 x1874668972727808/t8589944739(8589944739) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827151 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787827139.934484:0:94714:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94714:1874668972729088:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827139.934500:0:94714:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.934511:0:94714:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.934518:0:94714:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.934530:0:94714:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.934538:0:94714:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.934544:0:94714:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9c1a59013400 already unused, calling callback (ffffffffc1bbd990) 00010000:00010000:0.0:1787827139.934549:0:94714:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.934556:0:94714:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.934563:0:94714:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 5/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.934569:0:94714:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827139.934577:0:94714:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827139.934583:0:94714:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013400/0x531a24091f9887c5 lrc: 0/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 94714 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827139.934597:0:94714:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0xb, status=-17 req@ffff9c1a78860780 x1874668972729088/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827156 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827139.934610:0:94714:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -17 00010000:00010000:1.0:1787827140.024419:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827140.279601:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972729216:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827140.279606:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972729344:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827140.279709:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972729600:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827140.279714:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972729472:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827140.279774:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972729728:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827140.281919:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972729344:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827140.281938:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972729600:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827140.282103:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972729216:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827140.282117:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972729472:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827140.282127:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972729728:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827140.359888:0:94751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94751:1874668972729856:192.168.203.148@tcp:101:rm.0 00010000:00010000:1.0:1787827140.359910:0:94751:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827140.359922:0:94751:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827140.359934:0:94751:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827140.359944:0:94751:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827140.359955:0:94751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827140.359962:0:94751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827140.359969:0:94751:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827140.359980:0:94751:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a49d8e580 x1874668972729856/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827157 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827140.359992:0:94751:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827140.360003:0:94751:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827140.360010:0:94751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827140.360033:0:94751:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827140.360039:0:94751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9887cc 00000080:00010000:1.0:1787827140.360052:0:94751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827140.360055:0:94751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887cc from it ffffbe1909763b58 00010000:00010000:1.0:1787827140.360059:0:94751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827140.360067:0:94751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827140.360098:0:94751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827140.360108:0:94751:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=8 00000080:00200008:1.0:1787827140.360120:0:94751:0:(file.c:6313:ll_getattr_dentry()) START file f41i.sanityn:[0x200000402:0x1fb:0x0](ffff9c1a813f8908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787827140.360125:0:94751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1fb:0x0](ffff9c1a813f8908),name=f41i.sanityn 00000002:00010000:1.0:1787827140.360144:0:94751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1fb:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827140.360153:0:94751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827140.360163:0:94751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827140.360170:0:94751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827140.360174:0:94751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1fb:0x0](ffff9c1a813f8908) 00000080:00010000:1.0:1787827140.360177:0:94751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1fb:0x0](ffff9c1a813f8908) for lock 0x531a24091f9887b7 00000080:00010000:1.0:1787827140.360181:0:94751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887b7 from it ffffbe1909763cb8 00010000:00010000:1.0:1787827140.360183:0:94751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827140.360190:0:94751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827140.360240:0:94751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827140.360248:0:94751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827140.360254:0:94751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827140.360262:0:94751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827140.360268:0:94751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827140.360272:0:94751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827140.360274:0:94751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9887cc 00000080:00010000:1.0:1787827140.360278:0:94751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887cc from it ffffbe1909763c10 00010000:00010000:1.0:1787827140.360281:0:94751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827140.360288:0:94751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827140.360295:0:94751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827140.360302:0:94751:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=0 00000080:00200000:1.0:1787827140.360306:0:94751:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827140.360311:0:94751:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827140.360317:0:94751:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827140.360323:0:94751:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827140.360329:0:94751:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827140.360332:0:94751:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827140.360335:0:94751:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9887cc 00000080:00010000:1.0:1787827140.360338:0:94751:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887cc from it ffffbe1909763d68 00010000:00010000:1.0:1787827140.360341:0:94751:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827140.360516:0:94751:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827140.360525:0:94751:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:1.0:1787827140.360531:0:94751:0:(namei.c:2418:ll_unlink()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248) 00010000:00010000:1.0:1787827140.360556:0:94751:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827140.360577:0:94751:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827140.360587:0:94751:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x1fb:0x0]: losing layout lock 00010000:00010000:1.0:1787827140.360614:0:94751:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827140.360621:0:94751:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827140.360627:0:94751:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1787827140.360629:0:94751:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887b7 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1fb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb30e6 expref: -99 pid: 94712 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827140.360643:0:94751:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941200/0x531a24091f9887cc lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb3117 expref: -99 pid: 94751 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1787827140.360664:0:94751:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827140.360672:0:94751:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a79a40 x1874668972730112/t0(0) o36->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827140.360691:0:94751:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94751:1874668972730112:192.168.203.148@tcp:36:rm.0 00000100:00100000:1.0:1787827140.360760:0:94751:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1c00 going to sleep for 12 seconds 00000100:00100000:1.0:1787827140.363451:0:94751:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94751:1874668972730112:192.168.203.148@tcp:36:rm.0 00000080:00200000:1.0:1787827140.363498:0:94751:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x1fb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1787827140.363508:0:94751:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1fb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787827140.363515:0:94751:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x1fb:0x0](ffff9c1a813f8908) 00010000:00010000:0.0:1787827141.047517:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827143.095538:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827144.119482:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827145.399622:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2411:1874668972730752:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827145.399644:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972730880:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827145.399771:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972731008:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827145.399773:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972731136:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827145.399814:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972731392:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827145.399834:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972731264:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827145.399871:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972731520:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827145.403544:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972730880:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827145.403568:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972731136:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827145.403584:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972731392:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827145.403657:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972731264:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827145.403680:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972731520:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827145.403691:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2411:1874668972730752:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827145.403717:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972731008:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827146.074112:0:94817:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94817:1874668972730624:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827146.074133:0:94817:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.074147:0:94817:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.074165:0:94817:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.074175:0:94817:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.074184:0:94817:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.074192:0:94817:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.074205:0:94817:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827146.074212:0:94817:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a49d8f0c0 x1874668972730624/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827158 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827146.074222:0:94817:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.074242:0:94817:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827146.074248:0:94817:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827146.074273:0:94817:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827146.074279:0:94817:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f9887e1 00000080:00010000:1.0:1787827146.074284:0:94817:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:1.0:1787827146.074298:0:94817:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887e1 from it ffffbe19097ebbf8 00010000:00010000:1.0:1787827146.074305:0:94817:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.074319:0:94817:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827146.074336:0:94817:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 1 00000080:00200000:1.0:1787827146.074384:0:94817:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:1.0:1787827146.074402:0:94817:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827146.074419:0:94817:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827146.074435:0:94817:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827146.074442:0:94817:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827146.074449:0:94817:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:1.0:1787827146.074453:0:94817:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f9887e1 00000080:00010000:1.0:1787827146.074455:0:94817:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887e1 from it ffffbe19097ebb38 00010000:00010000:1.0:1787827146.074457:0:94817:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827146.074462:0:94817:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827146.074471:0:94817:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 3 00000080:00200000:1.0:1787827146.074477:0:94817:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), file ffff9c1a590cb980, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:1.0:1787827146.074485:0:94817:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), intent=open|creat 00000002:00010000:1.0:1787827146.074499:0:94817:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000403:0x1c01:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:1.0:1787827146.074506:0:94817:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827146.074557:0:94817:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827146.074568:0:94817:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827146.074573:0:94817:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941400/0x531a24091f9887e1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb3164 expref: -99 pid: 94817 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827146.074597:0:94817:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.074602:0:94817:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787827146.074609:0:94817:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a49bbf800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787827146.074615:0:94817:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a78f00 x1874668972731776/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787827146.074625:0:94817:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827146.074646:0:94817:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94817:1874668972731776:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827146.074758:0:94817:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1c00 going to sleep for 12 seconds 00000100:00100000:1.0:1787827146.076120:0:94817:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94817:1874668972731776:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827146.076135:0:94817:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.076145:0:94817:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.076152:0:94817:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.076158:0:94817:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.076166:0:94817:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.076173:0:94817:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9c1a46ef7400 already unused, calling callback (ffffffffc1bbd990) 00010000:00010000:1.0:1787827146.076178:0:94817:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.076185:0:94817:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.076193:0:94817:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 5/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.076201:0:94817:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1787827146.076207:0:94817:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.076218:0:94817:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f9887e8 lrc: 0/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 94817 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827146.076232:0:94817:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0xb, status=-17 req@ffff9c1a78a78f00 x1874668972731776/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827163 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1787827146.076243:0:94817:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -17 00010000:00010000:1.0:1787827146.167511:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1787827146.463675:0:94854:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827146.463714:0:94854:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827146.463768:0:94854:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94854 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.463778:0:94854:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94854 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.463788:0:94854:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94854 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827146.463810:0:94854:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94854:1874668972732160:192.168.203.148@tcp:101:rm.0 00000100:00100000:1.0:1787827146.463888:0:94854:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1a80 going to sleep for 12 seconds 00000100:00100000:1.0:1787827146.467211:0:94854:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94854:1874668972732160:192.168.203.148@tcp:101:rm.0 00010000:00010000:1.0:1787827146.467235:0:94854:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.467248:0:94854:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.467263:0:94854:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.467272:0:94854:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.467282:0:94854:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.467290:0:94854:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.467297:0:94854:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827146.467308:0:94854:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78a78f00 x1874668972732160/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827163 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827146.467320:0:94854:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.467331:0:94854:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827146.467337:0:94854:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827146.467532:0:94854:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827146.467539:0:94854:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9887ef 00000080:00010000:1.0:1787827146.467545:0:94854:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827146.467548:0:94854:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887ef from it ffffbe1909783b58 00010000:00010000:1.0:1787827146.467552:0:94854:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.467561:0:94854:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827146.467582:0:94854:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827146.467591:0:94854:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=8 00000080:00200008:1.0:1787827146.467601:0:94854:0:(file.c:6313:ll_getattr_dentry()) START file f41i.sanityn:[0x200000402:0x1fc:0x0](ffff9c1a813b6c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787827146.467605:0:94854:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1fc:0x0](ffff9c1a813b6c08),name=f41i.sanityn 00000002:00010000:1.0:1787827146.467621:0:94854:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1fc:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827146.467628:0:94854:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21600/0x531a24091f9887da lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb316b expref: -99 pid: 94815 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827146.467635:0:94854:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21600/0x531a24091f9887da lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb316b expref: -99 pid: 94815 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827146.467640:0:94854:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827146.467643:0:94854:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1fc:0x0](ffff9c1a813b6c08) 00000080:00010000:1.0:1787827146.467645:0:94854:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1fc:0x0](ffff9c1a813b6c08) for lock 0x531a24091f9887da 00000080:00010000:1.0:1787827146.467647:0:94854:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887da from it ffffbe1909783cb8 00010000:00010000:1.0:1787827146.467649:0:94854:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21600/0x531a24091f9887da lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb316b expref: -99 pid: 94815 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827146.467653:0:94854:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21600/0x531a24091f9887da lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb316b expref: -99 pid: 94815 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827146.467707:0:94854:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827146.467715:0:94854:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827146.467721:0:94854:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827146.467730:0:94854:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827146.467736:0:94854:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827146.467740:0:94854:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827146.467743:0:94854:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9887ef 00000080:00010000:1.0:1787827146.467746:0:94854:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887ef from it ffffbe1909783c10 00010000:00010000:1.0:1787827146.467749:0:94854:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827146.467754:0:94854:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827146.467759:0:94854:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827146.467764:0:94854:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=0 00000080:00200000:1.0:1787827146.467767:0:94854:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827146.467771:0:94854:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827146.467774:0:94854:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827146.467779:0:94854:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827146.467782:0:94854:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827146.467784:0:94854:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827146.467786:0:94854:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9887ef 00000080:00010000:1.0:1787827146.467788:0:94854:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887ef from it ffffbe1909783d68 00010000:00010000:1.0:1787827146.467790:0:94854:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827146.467794:0:94854:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827146.467797:0:94854:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:1.0:1787827146.467800:0:94854:0:(namei.c:2418:ll_unlink()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248) 00010000:00010000:1.0:1787827146.467817:0:94854:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827146.467835:0:94854:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21600/0x531a24091f9887da lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bb316b expref: -99 pid: 94815 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827146.467844:0:94854:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x1fc:0x0]: losing layout lock 00010000:00010000:1.0:1787827146.467871:0:94854:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21600/0x531a24091f9887da lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb316b expref: -99 pid: 94815 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827146.467877:0:94854:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827146.467883:0:94854:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1787827146.467885:0:94854:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21600/0x531a24091f9887da lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1fc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb316b expref: -99 pid: 94815 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827146.467898:0:94854:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887ef lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb319c expref: -99 pid: 94854 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1787827146.467918:0:94854:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827146.467924:0:94854:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a78f00 x1874668972732416/t0(0) o36->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827146.467944:0:94854:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94854:1874668972732416:192.168.203.148@tcp:36:rm.0 00000100:00100000:1.0:1787827146.468005:0:94854:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43ed1a80 going to sleep for 12 seconds 00000100:00100000:1.0:1787827146.469974:0:94854:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94854:1874668972732416:192.168.203.148@tcp:36:rm.0 00000080:00200000:1.0:1787827146.470018:0:94854:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x1fc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1787827146.470028:0:94854:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1fc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787827146.470035:0:94854:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x1fc:0x0](ffff9c1a813b6c08) 00000080:00200000:1.0:1787827146.924880:0:94918:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827146.924907:0:94918:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827146.924953:0:94918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94918 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.924961:0:94918:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94918 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.924967:0:94918:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94918 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827146.924985:0:94918:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94918:1874668972732544:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827146.925060:0:94918:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a436953c0 going to sleep for 12 seconds 00000100:00100000:1.0:1787827146.926395:0:94918:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94918:1874668972732544:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827146.926413:0:94918:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.926424:0:94918:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.926437:0:94918:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.926448:0:94918:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.926459:0:94918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.926467:0:94918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.926474:0:94918:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827146.926484:0:94918:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78a792c0 x1874668972732544/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827163 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827146.926498:0:94918:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.926509:0:94918:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827146.926514:0:94918:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827146.926537:0:94918:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827146.926544:0:94918:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9887f6 00000080:00010000:1.0:1787827146.926550:0:94918:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827146.926553:0:94918:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887f6 from it ffffbe19098cbbf8 00010000:00010000:1.0:1787827146.926557:0:94918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.926566:0:94918:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827146.926589:0:94918:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827146.926606:0:94918:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827146.926616:0:94918:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827146.926623:0:94918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827146.926633:0:94918:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827146.926640:0:94918:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827146.926645:0:94918:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827146.926648:0:94918:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f9887f6 00000080:00010000:1.0:1787827146.926651:0:94918:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887f6 from it ffffbe19098cbb38 00010000:00010000:1.0:1787827146.926655:0:94918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827146.926663:0:94918:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827146.926677:0:94918:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:1.0:1787827146.926682:0:94918:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), file ffff9c1a80964600, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:1.0:1787827146.926709:0:94918:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000002:00010000:1.0:1787827146.926728:0:94918:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000402:0x1fd:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:1.0:1787827146.926738:0:94918:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827146.926772:0:94918:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827146.926779:0:94918:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827146.926783:0:94918:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7400/0x531a24091f9887f6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb31c6 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827146.926810:0:94918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94918 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827146.926818:0:94918:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94918 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787827146.926828:0:94918:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787827146.926835:0:94918:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a792c0 x1874668972732800/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787827146.926843:0:94918:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94918 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827146.926857:0:94918:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94918:1874668972732800:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827146.926957:0:94918:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a436953c0 going to sleep for 12 seconds 00000080:00200000:1.0:1787827147.129064:0:94920:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:1.0:1787827147.129085:0:94920:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827147.129095:0:94920:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:1.0:1787827147.129133:0:94920:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f988804 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 94920 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827147.129140:0:94920:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f988804 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94920 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827147.129146:0:94920:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7400/0x531a24091f988804 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 94920 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827147.129165:0:94920:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:94920:1874668972732928:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827147.129234:0:94920:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43695e40 going to sleep for 12 seconds 00010000:00010000:1.0:1787827147.191572:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827148.215501:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827150.263572:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827151.287620:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827152.217679:0:94918:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9c1a46db5000: committing for last_committed 8589944744 gen 1 00000100:00100000:1.0:1787827152.217707:0:94918:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9c1a78a792c0 x1874668972732800/t8589944745(8589944745) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827163 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1787827152.217717:0:94918:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94918:1874668972732800:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827152.217739:0:94918:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827152.217761:0:94918:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827152.217770:0:94918:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827152.217790:0:94918:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827152.217809:0:94918:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827152.217833:0:94918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827152.217849:0:94918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827152.217870:0:94918:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827152.217889:0:94918:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9c1a78a792c0 x1874668972732800/t8589944745(8589944745) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827163 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787827152.217912:0:94918:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78a792c0 x1874668972732800/t8589944745(8589944745) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827163 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827152.217933:0:94918:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827152.217949:0:94918:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589935618/509 (0 0) 00000080:00200000:1.0:1787827152.217956:0:94918:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827152.217968:0:94918:0:(namei.c:1628:ll_create_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000080:00200000:1.0:1787827152.217995:0:94918:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x1fd:0x0](ffff9c1a813f8908) 00000080:00200000:1.0:1787827152.218001:0:94918:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x1fd:0x0], updating i_size 0 00000080:00200000:1.0:1787827152.218045:0:94918:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x1fd:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1787827152.218051:0:94918:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x1fd:0x0](ffff9c1a813f8908) 00000080:00010000:1.0:1787827152.218058:0:94918:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1fd:0x0](ffff9c1a813f8908) for lock 0x531a24091f9887fd 00000080:00010000:1.0:1787827152.218066:0:94918:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1fd:0x0](ffff9c1a813f8908) 00000080:00200008:1.0:1787827152.218076:0:94918:0:(file.c:1090:ll_file_open()) START file f41i.sanityn:[0x200000402:0x1fd:0x0](ffff9c1a813f8908), flags 100300 00000002:00100000:1.0:1787827152.218098:0:94918:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78a792c0 x1874668972732800/t8589944745(8589944745) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827163 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787827152.218138:0:94918:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887fd from it ffff9c1a52928240 00010000:00010000:1.0:1787827152.218146:0:94918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827152.218169:0:94918:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827152.218189:0:94918:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1fd:0x0](ffff9c1a813f8908), inode mode 81a4 mask 40 00000080:00200008:1.0:1787827152.218219:0:94918:0:(file.c:479:ll_file_release()) START file f41i.sanityn:[0x200000402:0x1fd:0x0](ffff9c1a813f8908), flags 100000 00000002:00100000:1.0:1787827152.218256:0:94918:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9c1a78a792c0 x1874668972732800/t8589944745(8589944745) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827163 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787827152.218266:0:94918:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787827152.218271:0:94918:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a78f00 x1874668972734336/t0(0) o35->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787827152.218285:0:94918:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94918:1874668972734336:192.168.203.148@tcp:35:multiop.0 00000100:00100000:1.0:1787827152.218405:0:94918:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a436953c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787827152.219766:0:94918:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:94918:1874668972734336:192.168.203.148@tcp:35:multiop.0 00000002:00100000:1.0:1787827152.219787:0:94918:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9c1a78a792c0 x1874668972732800/t8589944745(8589944745) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827163 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1787827152.311499:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787827152.653924:0:94957:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827152.653951:0:94957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827152.654005:0:94957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 94957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827152.654016:0:94957:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827152.654026:0:94957:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 94957 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827152.654048:0:94957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94957:1874668972734464:192.168.203.148@tcp:101:rm.0 00000100:00100000:0.0:1787827152.654135:0:94957:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146000 going to sleep for 12 seconds 00000100:00100000:0.0:1787827152.655860:0:94957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94957:1874668972734464:192.168.203.148@tcp:101:rm.0 00010000:00010000:0.0:1787827152.655879:0:94957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827152.655897:0:94957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827152.655908:0:94957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827152.655918:0:94957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827152.655929:0:94957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827152.655936:0:94957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827152.655943:0:94957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827152.655953:0:94957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78861a40 x1874668972734464/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827169 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827152.655964:0:94957:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827152.655975:0:94957:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827152.655981:0:94957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827152.656005:0:94957:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827152.656012:0:94957:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988812 00000080:00010000:0.0:1787827152.656018:0:94957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827152.656021:0:94957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988812 from it ffffbe190986bb58 00010000:00010000:0.0:1787827152.656025:0:94957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827152.656033:0:94957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827152.656054:0:94957:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827152.656064:0:94957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=8 00000080:00200008:0.0:1787827152.656074:0:94957:0:(file.c:6313:ll_getattr_dentry()) START file f41i.sanityn:[0x200000402:0x1fd:0x0](ffff9c1a813f8908), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827152.656079:0:94957:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1fd:0x0](ffff9c1a813f8908),name=f41i.sanityn 00000002:00010000:0.0:1787827152.656096:0:94957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1fd:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827152.656104:0:94957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827152.656113:0:94957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827152.656120:0:94957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827152.656124:0:94957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1fd:0x0](ffff9c1a813f8908) 00000080:00010000:0.0:1787827152.656127:0:94957:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1fd:0x0](ffff9c1a813f8908) for lock 0x531a24091f9887fd 00000080:00010000:0.0:1787827152.656130:0:94957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f9887fd from it ffffbe190986bcb8 00010000:00010000:0.0:1787827152.656134:0:94957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827152.656142:0:94957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827152.656194:0:94957:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827152.656202:0:94957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827152.656209:0:94957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827152.656216:0:94957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827152.656222:0:94957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827152.656234:0:94957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827152.656237:0:94957:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988812 00000080:00010000:0.0:1787827152.656241:0:94957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988812 from it ffffbe190986bc10 00010000:00010000:0.0:1787827152.656245:0:94957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827152.656252:0:94957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827152.656261:0:94957:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827152.656267:0:94957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=0 00000080:00200000:0.0:1787827152.656271:0:94957:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827152.656277:0:94957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827152.656283:0:94957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827152.656291:0:94957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827152.656303:0:94957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827152.656307:0:94957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827152.656309:0:94957:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988812 00000080:00010000:0.0:1787827152.656312:0:94957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988812 from it ffffbe190986bd68 00010000:00010000:0.0:1787827152.656315:0:94957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827152.656332:0:94957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827152.656341:0:94957:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:0.0:1787827152.656371:0:94957:0:(namei.c:2418:ll_unlink()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248) 00010000:00010000:0.0:1787827152.656396:0:94957:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827152.656417:0:94957:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827152.656428:0:94957:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x1fd:0x0]: losing layout lock 00010000:00010000:0.0:1787827152.656452:0:94957:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827152.656460:0:94957:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827152.656466:0:94957:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787827152.656469:0:94957:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef7800/0x531a24091f9887fd lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1fd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb31f0 expref: -99 pid: 94918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827152.656483:0:94957:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59012c00/0x531a24091f988812 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb3221 expref: -99 pid: 94957 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787827152.656514:0:94957:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787827152.656522:0:94957:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78861a40 x1874668972734720/t0(0) o36->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827152.656542:0:94957:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94957:1874668972734720:192.168.203.148@tcp:36:rm.0 00000100:00100000:0.0:1787827152.656610:0:94957:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146000 going to sleep for 12 seconds 00000100:00100000:0.0:1787827152.659150:0:94957:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:94957:1874668972734720:192.168.203.148@tcp:36:rm.0 00000080:00200000:0.0:1787827152.659236:0:94957:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x1fd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827152.659247:0:94957:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1fd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827152.659254:0:94957:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x1fd:0x0](ffff9c1a813f8908) 00010000:00010000:3.0:1787827153.149728:0:95021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21400/0x531a24091f988819 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb324b expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827153.149739:0:95021:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21400/0x531a24091f988819 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb324b expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827153.149742:0:95021:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1787827153.149745:0:95021:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:3.0:1787827153.149747:0:95021:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988819 00000080:00010000:3.0:1787827153.149749:0:95021:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988819 from it ffffbe19097e3b38 00010000:00010000:3.0:1787827153.149752:0:95021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21400/0x531a24091f988819 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb324b expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1787827153.149759:0:95021:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21400/0x531a24091f988819 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb324b expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1787827153.149766:0:95021:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:3.0:1787827153.149771:0:95021:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), file ffff9c1a4552b980, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:3.0:1787827153.149778:0:95021:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000002:00010000:3.0:1787827153.149795:0:95021:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000402:0x1fe:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:3.0:1787827153.149806:0:95021:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21400/0x531a24091f988819 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb324b expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827153.149840:0:95021:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21400/0x531a24091f988819 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb324b expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827153.149844:0:95021:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1787827153.149846:0:95021:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21400/0x531a24091f988819 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb324b expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1787827153.149880:0:95021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 95021 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827153.149887:0:95021:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95021 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1787827153.149897:0:95021:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1787827153.149903:0:95021:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a7895d2c0 x1874668972735104/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1787827153.149913:0:95021:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95021 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787827153.149929:0:95021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95021:1874668972735104:192.168.203.148@tcp:101:multiop.0 00000100:00100000:3.0:1787827153.150044:0:95021:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a6014ce40 going to sleep for 12 seconds 00010000:00010000:1.0:1787827153.335832:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827154.359574:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827155.383526:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827155.639788:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972735360:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827155.639798:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972735488:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827155.639902:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972735616:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827155.639909:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972735744:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827155.639938:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972736000:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827155.639948:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972735872:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827155.639986:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972736128:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827155.641228:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972735360:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827155.641529:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972735488:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827155.641552:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972735616:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827155.641555:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972735744:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827155.641576:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972735872:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827155.641881:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972736128:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827155.641898:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972736000:192.168.203.148@tcp:400:kworker.0 00010000:00010000:3.0:1787827156.407564:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827157.431598:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827158.363567:0:95021:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9c1a46db5000: committing for last_committed 8589944747 gen 1 00000100:00100000:3.0:1787827158.363603:0:95021:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9c1a7895d2c0 x1874668972735104/t8589944748(8589944748) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827170 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1787827158.363629:0:95021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95021:1874668972735104:192.168.203.148@tcp:101:multiop.0 00010000:00010000:3.0:1787827158.363650:0:95021:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827158.363681:0:95021:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827158.363690:0:95021:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827158.363701:0:95021:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827158.363710:0:95021:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827158.363722:0:95021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827158.363731:0:95021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827158.363739:0:95021:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1787827158.363749:0:95021:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9c1a7895d2c0 x1874668972735104/t8589944748(8589944748) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827170 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1787827158.363762:0:95021:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a7895d2c0 x1874668972735104/t8589944748(8589944748) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827170 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1787827158.363774:0:95021:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827158.363784:0:95021:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589935618/510 (0 0) 00000080:00200000:3.0:1787827158.363791:0:95021:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1787827158.363803:0:95021:0:(namei.c:1628:ll_create_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000080:00200000:3.0:1787827158.363869:0:95021:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x1fe:0x0](ffff9c1a813b0908) 00000080:00200000:3.0:1787827158.363877:0:95021:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x1fe:0x0], updating i_size 0 00000080:00200000:3.0:1787827158.363909:0:95021:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x1fe:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1787827158.363914:0:95021:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x1fe:0x0](ffff9c1a813b0908) 00000080:00010000:3.0:1787827158.363917:0:95021:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1fe:0x0](ffff9c1a813b0908) for lock 0x531a24091f988820 00000080:00010000:3.0:1787827158.363927:0:95021:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1fe:0x0](ffff9c1a813b0908) 00000080:00200008:3.0:1787827158.363937:0:95021:0:(file.c:1090:ll_file_open()) START file f41i.sanityn:[0x200000402:0x1fe:0x0](ffff9c1a813b0908), flags 100300 00000002:00100000:3.0:1787827158.363953:0:95021:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a7895d2c0 x1874668972735104/t8589944748(8589944748) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827170 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1787827158.363981:0:95021:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988820 from it ffff9c1a60033f60 00010000:00010000:3.0:1787827158.363984:0:95021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827158.363990:0:95021:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1787827158.364008:0:95021:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1fe:0x0](ffff9c1a813b0908), inode mode 81a4 mask 40 00000080:00200008:3.0:1787827158.364037:0:95021:0:(file.c:479:ll_file_release()) START file f41i.sanityn:[0x200000402:0x1fe:0x0](ffff9c1a813b0908), flags 100000 00000002:00100000:3.0:1787827158.364076:0:95021:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9c1a7895d2c0 x1874668972735104/t8589944748(8589944748) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827170 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1787827158.364084:0:95021:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1787827158.364087:0:95021:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a7895e580 x1874668972736384/t0(0) o35->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1787827158.364103:0:95021:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95021:1874668972736384:192.168.203.148@tcp:35:multiop.0 00000100:00100000:3.0:1787827158.364186:0:95021:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a6014ce40 going to sleep for 11 seconds 00000100:00100000:1.0:1787827158.364402:0:95023:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:95023:1874668972735232:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827158.364418:0:95023:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.364429:0:95023:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.364439:0:95023:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.364449:0:95023:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.364459:0:95023:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.364466:0:95023:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.364473:0:95023:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827158.364483:0:95023:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a49d8d680 x1874668972735232/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827170 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827158.364494:0:95023:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.364503:0:95023:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827158.364508:0:95023:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827158.364542:0:95023:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827158.364548:0:95023:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f988827 00000080:00010000:1.0:1787827158.364553:0:95023:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:1.0:1787827158.364556:0:95023:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988827 from it ffffbe19099cbbf8 00010000:00010000:1.0:1787827158.364560:0:95023:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.364569:0:95023:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827158.364590:0:95023:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 1 00000080:00200000:1.0:1787827158.364605:0:95023:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:1.0:1787827158.364630:0:95023:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827158.364638:0:95023:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827158.364647:0:95023:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827158.364653:0:95023:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827158.364658:0:95023:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:1.0:1787827158.364660:0:95023:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f988827 00000080:00010000:1.0:1787827158.364663:0:95023:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988827 from it ffffbe19099cbb38 00010000:00010000:1.0:1787827158.364666:0:95023:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827158.364673:0:95023:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827158.364680:0:95023:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 3 00000080:00200000:1.0:1787827158.364684:0:95023:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), file ffff9c1a448f5980, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:1.0:1787827158.364690:0:95023:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), intent=open|creat 00000002:00010000:1.0:1787827158.364705:0:95023:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000403:0x1c03:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:1.0:1787827158.364715:0:95023:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827158.364759:0:95023:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827158.364765:0:95023:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827158.364768:0:95023:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7a941200/0x531a24091f988827 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb326e expref: -99 pid: 95023 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827158.364789:0:95023:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.364795:0:95023:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787827158.364802:0:95023:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a49bbf800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787827158.364807:0:95023:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a783c0 x1874668972736640/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787827158.364816:0:95023:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827158.364828:0:95023:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:95023:1874668972736640:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827158.364911:0:95023:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a436953c0 going to sleep for 12 seconds 00000100:00100000:3.0:1787827158.365261:0:95021:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95021:1874668972736384:192.168.203.148@tcp:35:multiop.0 00000002:00100000:3.0:1787827158.365280:0:95021:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9c1a7895d2c0 x1874668972735104/t8589944748(8589944748) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827170 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1787827158.367146:0:95023:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:95023:1874668972736640:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827158.367158:0:95023:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.367167:0:95023:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.367174:0:95023:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.367179:0:95023:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.367184:0:95023:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.367189:0:95023:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9c1a46ef6000 already unused, calling callback (ffffffffc1bbd990) 00010000:00010000:1.0:1787827158.367193:0:95023:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.367199:0:95023:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.367206:0:95023:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 5/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.367212:0:95023:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1787827158.367218:0:95023:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.367225:0:95023:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef6000/0x531a24091f98882e lrc: 0/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 95023 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827158.367240:0:95023:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0xb, status=-17 req@ffff9c1a78a783c0 x1874668972736640/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827175 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1787827158.367252:0:95023:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -17 00010000:00010000:3.0:1787827158.455493:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827158.785659:0:95060:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95060:1874668972736768:192.168.203.148@tcp:101:rm.0 00010000:00010000:1.0:1787827158.785682:0:95060:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.785695:0:95060:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.785707:0:95060:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.785718:0:95060:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.785729:0:95060:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.785738:0:95060:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.785746:0:95060:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827158.785758:0:95060:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a49d8c780 x1874668972736768/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827175 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827158.785772:0:95060:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.785785:0:95060:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827158.785791:0:95060:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827158.785816:0:95060:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827158.785821:0:95060:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988835 00000080:00010000:1.0:1787827158.785826:0:95060:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827158.785830:0:95060:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988835 from it ffffbe1909a13b58 00010000:00010000:1.0:1787827158.785834:0:95060:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827158.785842:0:95060:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827158.785877:0:95060:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827158.785888:0:95060:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=8 00000080:00200008:1.0:1787827158.785898:0:95060:0:(file.c:6313:ll_getattr_dentry()) START file f41i.sanityn:[0x200000402:0x1fe:0x0](ffff9c1a813b0908), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787827158.785903:0:95060:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1fe:0x0](ffff9c1a813b0908),name=f41i.sanityn 00000002:00010000:1.0:1787827158.785921:0:95060:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1fe:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827158.785931:0:95060:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827158.785942:0:95060:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827158.785949:0:95060:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827158.785953:0:95060:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1fe:0x0](ffff9c1a813b0908) 00000080:00010000:1.0:1787827158.785956:0:95060:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1fe:0x0](ffff9c1a813b0908) for lock 0x531a24091f988820 00000080:00010000:1.0:1787827158.785960:0:95060:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988820 from it ffffbe1909a13cb8 00010000:00010000:1.0:1787827158.785963:0:95060:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827158.785977:0:95060:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827158.786019:0:95060:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827158.786026:0:95060:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827158.786032:0:95060:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827158.786041:0:95060:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827158.786047:0:95060:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827158.786051:0:95060:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827158.786053:0:95060:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988835 00000080:00010000:1.0:1787827158.786057:0:95060:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988835 from it ffffbe1909a13c10 00010000:00010000:1.0:1787827158.786060:0:95060:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827158.786067:0:95060:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827158.786074:0:95060:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827158.786079:0:95060:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=0 00000080:00200000:1.0:1787827158.786082:0:95060:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827158.786087:0:95060:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827158.786093:0:95060:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827158.786101:0:95060:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827158.786108:0:95060:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827158.786112:0:95060:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827158.786114:0:95060:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988835 00000080:00010000:1.0:1787827158.786118:0:95060:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988835 from it ffffbe1909a13d68 00010000:00010000:1.0:1787827158.786121:0:95060:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827158.786129:0:95060:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827158.786137:0:95060:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:1.0:1787827158.786143:0:95060:0:(namei.c:2418:ll_unlink()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248) 00010000:00010000:1.0:1787827158.786167:0:95060:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827158.786189:0:95060:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827158.786200:0:95060:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x1fe:0x0]: losing layout lock 00010000:00010000:1.0:1787827158.786228:0:95060:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827158.786237:0:95060:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827158.786249:0:95060:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1787827158.786252:0:95060:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a43e21a00/0x531a24091f988820 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1fe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb3275 expref: -99 pid: 95021 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827158.786268:0:95060:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941c00/0x531a24091f988835 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb32a6 expref: -99 pid: 95060 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1787827158.786285:0:95060:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827158.786293:0:95060:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a783c0 x1874668972737024/t0(0) o36->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827158.786312:0:95060:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95060:1874668972737024:192.168.203.148@tcp:36:rm.0 00000100:00100000:1.0:1787827158.786556:0:95060:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a436959c0 going to sleep for 12 seconds 00000100:00100000:1.0:1787827158.789400:0:95060:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95060:1874668972737024:192.168.203.148@tcp:36:rm.0 00000080:00200000:1.0:1787827158.789444:0:95060:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x1fe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1787827158.789453:0:95060:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1fe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787827158.789458:0:95060:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x1fe:0x0](ffff9c1a813b0908) 00000080:00200000:0.0:1787827159.303468:0:95125:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827159.303494:0:95125:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827159.303549:0:95125:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 95125 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.303560:0:95125:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95125 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.303569:0:95125:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95125 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827159.303594:0:95125:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95125:1874668972737152:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827159.303702:0:95125:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d1463c0 going to sleep for 12 seconds 00000100:00100000:0.0:1787827159.305124:0:95125:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95125:1874668972737152:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827159.305143:0:95125:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.305154:0:95125:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.305167:0:95125:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.305177:0:95125:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.305193:0:95125:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.305201:0:95125:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.305208:0:95125:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827159.305218:0:95125:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78861680 x1874668972737152/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827176 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827159.305227:0:95125:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.305235:0:95125:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827159.305239:0:95125:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827159.305259:0:95125:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827159.305265:0:95125:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98883c 00000080:00010000:0.0:1787827159.305270:0:95125:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827159.305273:0:95125:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98883c from it ffffbe1909a63bf8 00010000:00010000:0.0:1787827159.305276:0:95125:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.305284:0:95125:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827159.305309:0:95125:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827159.305326:0:95125:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827159.305336:0:95125:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827159.305517:0:95125:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827159.305834:0:95125:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827159.305842:0:95125:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827159.305847:0:95125:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827159.305850:0:95125:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98883c 00000080:00010000:0.0:1787827159.305854:0:95125:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98883c from it ffffbe1909a63b38 00010000:00010000:0.0:1787827159.305857:0:95125:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827159.305864:0:95125:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827159.305872:0:95125:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:0.0:1787827159.305877:0:95125:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), file ffff9c1a602ee900, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:0.0:1787827159.305885:0:95125:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000002:00010000:0.0:1787827159.305902:0:95125:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000402:0x1ff:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:0.0:1787827159.305912:0:95125:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827159.305950:0:95125:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827159.305957:0:95125:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827159.305961:0:95125:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013000/0x531a24091f98883c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb32d0 expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827159.305987:0:95125:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 95125 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.305994:0:95125:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95125 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827159.306005:0:95125:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1787827159.306012:0:95125:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78861680 x1874668972737408/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827159.306020:0:95125:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95125 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827159.306034:0:95125:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95125:1874668972737408:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827159.306132:0:95125:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d1463c0 going to sleep for 12 seconds 00010000:00010000:1.0:1787827159.479501:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787827159.508103:0:95128:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:0.0:1787827159.508131:0:95128:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827159.508145:0:95128:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:0.0:1787827159.508194:0:95128:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.508204:0:95128:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827159.508213:0:95128:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827159.508237:0:95128:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:95128:1874668972737536:192.168.203.148@tcp:101:multiop.0 00000100:00100000:0.0:1787827159.508332:0:95128:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146e40 going to sleep for 12 seconds 00010000:00010000:1.0:1787827160.503574:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827160.759622:0:2414:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7127393f-350f-49ff-88de-bcfd4bafe057:2414:1874668972737792:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827160.759703:0:2414:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2414:1874668972738048:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827160.759738:0:2414:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2414:1874668972738304:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827160.764790:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2413:1874668972737664:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827160.764817:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7127393f-350f-49ff-88de-bcfd4bafe057:2413:1874668972737920:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827160.764828:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972738176:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827160.764870:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972738432:192.168.203.148@tcp:400:kworker.0 00010000:00010000:1.0:1787827161.527614:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827162.551517:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827163.575705:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827164.505899:0:95128:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:95128:1874668972737536:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827164.505925:0:95125:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9c1a46db5000: committing for last_committed 8589944750 gen 1 00010000:00010000:0.0:1787827164.505927:0:95128:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.505941:0:95128:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.505953:0:95128:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827164.505957:0:95125:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9c1a78861680 x1874668972737408/t8589944751(8589944751) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827176 ref 3 fl Rpc:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787827164.505966:0:95128:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.505986:0:95128:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827164.505988:0:95125:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95125:1874668972737408:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827164.506006:0:95128:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827164.506007:0:95125:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.506025:0:95128:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827164.506045:0:95128:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a788612c0 x1874668972737536/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827176 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1787827164.506056:0:95125:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1787827164.506058:0:95128:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827164.506074:0:95125:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.506080:0:95128:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 4 res 8589934599/1 (0 0) 00010000:00010000:1.0:1787827164.506084:0:95125:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827164.506087:0:95128:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00010000:00010000:1.0:1787827164.506101:0:95125:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827164.506110:0:95125:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827164.506111:0:95128:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827164.506120:0:95128:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f98884a 00010000:00010000:1.0:1787827164.506122:0:95125:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1787827164.506129:0:95128:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00010000:00010000:1.0:1787827164.506132:0:95125:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1787827164.506134:0:95128:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98884a from it ffffbe1909ab3bf8 00010000:00010000:0.0:1787827164.506141:0:95128:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827164.506146:0:95125:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9c1a78861680 x1874668972737408/t8589944751(8589944751) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827176 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1787827164.506157:0:95128:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1787827164.506163:0:95125:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78861680 x1874668972737408/t8589944751(8589944751) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827176 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827164.506176:0:95125:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1787827164.506182:0:95128:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 1 00010000:00010000:1.0:1787827164.506186:0:95125:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589935618/511 (0 0) 00000080:00200000:1.0:1787827164.506193:0:95125:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827164.506203:0:95125:0:(namei.c:1628:ll_create_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000080:00200000:0.0:1787827164.506215:0:95128:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000080:00200000:1.0:1787827164.506226:0:95125:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x1ff:0x0](ffff9c1a813fdb88) 00000002:00010000:0.0:1787827164.506231:0:95128:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000080:00200000:1.0:1787827164.506232:0:95125:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x1ff:0x0], updating i_size 0 00010000:00010000:0.0:1787827164.506244:0:95128:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827164.506255:0:95128:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827164.506264:0:95125:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x1ff:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00200000:0.0:1787827164.506268:0:95128:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827164.506270:0:95125:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x1ff:0x0](ffff9c1a813fdb88) 00000080:00010000:0.0:1787827164.506274:0:95128:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) 00000080:00010000:1.0:1787827164.506275:0:95125:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1ff:0x0](ffff9c1a813fdb88) for lock 0x531a24091f988843 00000080:00010000:0.0:1787827164.506281:0:95128:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d900c8) for lock 0x531a24091f98884a 00000080:00010000:1.0:1787827164.506286:0:95125:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1ff:0x0](ffff9c1a813fdb88) 00000080:00010000:0.0:1787827164.506287:0:95128:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98884a from it ffffbe1909ab3b38 00010000:00010000:0.0:1787827164.506291:0:95128:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827164.506298:0:95128:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827164.506308:0:95128:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8), inode mode 41ff mask 3 00000080:00200008:1.0:1787827164.506310:0:95125:0:(file.c:1090:ll_file_open()) START file f41i.sanityn:[0x200000402:0x1ff:0x0](ffff9c1a813fdb88), flags 100300 00000080:00200000:0.0:1787827164.506314:0:95128:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), file ffff9c1a49677680, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:0.0:1787827164.506323:0:95128:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d900c8), intent=open|creat 00000002:00100000:1.0:1787827164.506332:0:95125:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78861680 x1874668972737408/t8589944751(8589944751) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827176 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827164.506341:0:95128:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000403:0x1c04:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:0.0:1787827164.506383:0:95128:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1787827164.506438:0:95125:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988843 from it ffff9c1a7a8ed540 00010000:00010000:0.0:1787827164.506439:0:95128:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827164.506446:0:95125:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.506450:0:95128:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1787827164.506456:0:95128:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59013000/0x531a24091f98884a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb32f3 expref: -99 pid: 95128 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827164.506460:0:95125:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827164.506483:0:95125:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1ff:0x0](ffff9c1a813fdb88), inode mode 81a4 mask 40 00010000:00010000:0.0:1787827164.506484:0:95128:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.506497:0:95128:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1787827164.506507:0:95128:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a49bbf800: modify RPC slot 1 is allocated opc 101, max 7 00000080:00200008:1.0:1787827164.506511:0:95125:0:(file.c:479:ll_file_release()) START file f41i.sanityn:[0x200000402:0x1ff:0x0](ffff9c1a813fdb88), flags 100000 00000100:00100000:0.0:1787827164.506514:0:95128:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a788612c0 x1874668972738688/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1787827164.506524:0:95128:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827164.506541:0:95128:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:95128:1874668972738688:192.168.203.148@tcp:101:multiop.0 00000002:00100000:1.0:1787827164.506559:0:95125:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9c1a78861680 x1874668972737408/t8589944751(8589944751) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827176 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787827164.506583:0:95125:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787827164.506588:0:95125:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a783c0 x1874668972738944/t0(0) o35->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787827164.506603:0:95125:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95125:1874668972738944:192.168.203.148@tcp:35:multiop.0 00000100:00100000:0.0:1787827164.506683:0:95128:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146e40 going to sleep for 12 seconds 00000100:00100000:1.0:1787827164.506683:0:95125:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43695480 going to sleep for 11 seconds 00000100:00100000:1.0:1787827164.507553:0:95125:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95125:1874668972738944:192.168.203.148@tcp:35:multiop.0 00000002:00100000:1.0:1787827164.507580:0:95125:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9c1a78861680 x1874668972737408/t8589944751(8589944751) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827176 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1787827164.507892:0:95128:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:95128:1874668972738688:192.168.203.148@tcp:101:multiop.0 00010000:00010000:0.0:1787827164.507903:0:95128:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.507913:0:95128:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.507920:0:95128:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.507926:0:95128:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.507934:0:95128:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.507940:0:95128:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9c1a59012800 already unused, calling callback (ffffffffc1bbd990) 00010000:00010000:0.0:1787827164.507945:0:95128:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.507952:0:95128:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 4/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.507968:0:95128:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 5/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.507975:0:95128:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1787827164.507983:0:95128:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 3/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.507990:0:95128:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a59012800/0x531a24091f988851 lrc: 0/0,0 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 95128 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827164.508005:0:95128:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0xb, status=-17 req@ffff9c1a788612c0 x1874668972738688/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827181 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1787827164.508016:0:95128:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc -17 00010000:00010000:0.0:1787827164.600470:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1787827164.899436:0:95165:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827164.899462:0:95165:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827164.899514:0:95165:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 95165 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.899523:0:95165:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95165 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.899533:0:95165:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95165 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827164.899557:0:95165:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95165:1874668972739072:192.168.203.148@tcp:101:rm.0 00000100:00100000:0.0:1787827164.899654:0:95165:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146e40 going to sleep for 12 seconds 00000100:00100000:0.0:1787827164.901040:0:95165:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95165:1874668972739072:192.168.203.148@tcp:101:rm.0 00010000:00010000:0.0:1787827164.901053:0:95165:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.901062:0:95165:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.901073:0:95165:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.901084:0:95165:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.901095:0:95165:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.901103:0:95165:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.901111:0:95165:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827164.901120:0:95165:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78863840 x1874668972739072/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827181 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827164.901133:0:95165:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.901148:0:95165:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827164.901153:0:95165:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827164.901176:0:95165:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827164.901182:0:95165:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988858 00000080:00010000:0.0:1787827164.901186:0:95165:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827164.901189:0:95165:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988858 from it ffffbe1909ab3b58 00010000:00010000:0.0:1787827164.901193:0:95165:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827164.901210:0:95165:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827164.901230:0:95165:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827164.901239:0:95165:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=8 00000080:00200008:0.0:1787827164.901248:0:95165:0:(file.c:6313:ll_getattr_dentry()) START file f41i.sanityn:[0x200000402:0x1ff:0x0](ffff9c1a813fdb88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827164.901253:0:95165:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1ff:0x0](ffff9c1a813fdb88),name=f41i.sanityn 00000002:00010000:0.0:1787827164.901266:0:95165:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1ff:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827164.901275:0:95165:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827164.901285:0:95165:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827164.901291:0:95165:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827164.901295:0:95165:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1ff:0x0](ffff9c1a813fdb88) 00000080:00010000:0.0:1787827164.901298:0:95165:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1ff:0x0](ffff9c1a813fdb88) for lock 0x531a24091f988843 00000080:00010000:0.0:1787827164.901303:0:95165:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988843 from it ffffbe1909ab3cb8 00010000:00010000:0.0:1787827164.901306:0:95165:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827164.901314:0:95165:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827164.901361:0:95165:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827164.901370:0:95165:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827164.901376:0:95165:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827164.901384:0:95165:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827164.901390:0:95165:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827164.901394:0:95165:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827164.901397:0:95165:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988858 00000080:00010000:0.0:1787827164.901400:0:95165:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988858 from it ffffbe1909ab3c10 00010000:00010000:0.0:1787827164.901403:0:95165:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827164.901411:0:95165:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827164.901423:0:95165:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827164.901428:0:95165:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=0 00000080:00200000:0.0:1787827164.901431:0:95165:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827164.901435:0:95165:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827164.901439:0:95165:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827164.901444:0:95165:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827164.901448:0:95165:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827164.901450:0:95165:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827164.901452:0:95165:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f988858 00000080:00010000:0.0:1787827164.901454:0:95165:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988858 from it ffffbe1909ab3d68 00010000:00010000:0.0:1787827164.901455:0:95165:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827164.901460:0:95165:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827164.901464:0:95165:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:0.0:1787827164.901468:0:95165:0:(namei.c:2418:ll_unlink()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248) 00010000:00010000:0.0:1787827164.901484:0:95165:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827164.901498:0:95165:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827164.901504:0:95165:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x1ff:0x0]: losing layout lock 00010000:00010000:0.0:1787827164.901521:0:95165:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827164.901529:0:95165:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827164.901533:0:95165:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787827164.901534:0:95165:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013800/0x531a24091f988843 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1ff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb32fa expref: -99 pid: 95125 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827164.901544:0:95165:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f988858 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb332b expref: -99 pid: 95165 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787827164.901556:0:95165:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787827164.901560:0:95165:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78863840 x1874668972739328/t0(0) o36->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827164.901586:0:95165:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95165:1874668972739328:192.168.203.148@tcp:36:rm.0 00000100:00100000:0.0:1787827164.901631:0:95165:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146e40 going to sleep for 12 seconds 00000100:00100000:0.0:1787827164.903402:0:95165:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95165:1874668972739328:192.168.203.148@tcp:36:rm.0 00000080:00200000:0.0:1787827164.903441:0:95165:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x1ff:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827164.903448:0:95165:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1ff:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827164.903453:0:95165:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x1ff:0x0](ffff9c1a813fdb88) 00000100:00100000:1.0:1787827165.348618:0:95229:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95229:1874668972739456:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827165.348643:0:95229:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827165.348652:0:95229:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827165.348661:0:95229:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827165.348667:0:95229:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827165.348674:0:95229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827165.348680:0:95229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827165.348685:0:95229:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827165.348692:0:95229:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a49d8da40 x1874668972739456/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827182 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827165.348702:0:95229:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827165.348710:0:95229:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1787827165.348715:0:95229:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1787827165.348735:0:95229:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1787827165.348739:0:95229:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98885f 00000080:00010000:1.0:1787827165.348742:0:95229:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827165.348744:0:95229:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98885f from it ffffbe1909ba3bf8 00010000:00010000:1.0:1787827165.348746:0:95229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827165.348752:0:95229:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827165.348768:0:95229:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:1.0:1787827165.348793:0:95229:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:1.0:1787827165.348805:0:95229:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827165.348813:0:95229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827165.348822:0:95229:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827165.348834:0:95229:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827165.348843:0:95229:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:1.0:1787827165.348846:0:95229:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98885f 00000080:00010000:1.0:1787827165.348849:0:95229:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98885f from it ffffbe1909ba3b38 00010000:00010000:1.0:1787827165.348853:0:95229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827165.348861:0:95229:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827165.348869:0:95229:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:1.0:1787827165.348874:0:95229:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), file ffff9c1a590d0f00, open_flags 80c0, mode 81b6 opened 0 00000080:00200000:1.0:1787827165.348883:0:95229:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000002:00010000:1.0:1787827165.348900:0:95229:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f41i.sanityn,[0x200000402:0x200:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:1.0:1787827165.348909:0:95229:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827165.348947:0:95229:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827165.348954:0:95229:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827165.348957:0:95229:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a7a941800/0x531a24091f98885f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb3355 expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827165.348988:0:95229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 95229 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827165.348999:0:95229:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95229 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1787827165.349009:0:95229:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1787827165.349015:0:95229:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a7ad00 x1874668972739712/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1787827165.349024:0:95229:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95229 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827165.349042:0:95229:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7ad00 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95229:1874668972739712:192.168.203.148@tcp:101:multiop.0 00000100:00100000:1.0:1787827165.349151:0:95229:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43695300 going to sleep for 12 seconds 00010000:00010000:3.0:1787827165.623547:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827165.879573:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2411:1874668972739968:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827165.879579:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972740096:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827165.879659:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972740224:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827165.879659:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972740352:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827165.879686:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972740608:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827165.879699:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972740480:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827165.881235:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2411:1874668972739968:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827165.881235:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972740096:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827165.881258:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972740224:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827165.881262:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972740352:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827165.881280:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972740608:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827165.881285:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972740480:192.168.203.148@tcp:400:kworker.0 00010000:00010000:1.0:1787827166.647515:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827167.671502:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827168.695657:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827169.719509:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827170.650213:0:95229:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7ad00 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95229:1874668972739712:192.168.203.148@tcp:101:multiop.0 00010000:00010000:1.0:1787827170.650239:0:95229:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827170.650275:0:95229:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x200:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827170.650286:0:95229:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827170.650317:0:95229:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827170.650344:0:95229:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827170.650398:0:95229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827170.650413:0:95229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827170.650426:0:95229:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827170.650439:0:95229:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9c1a78a7ad00 x1874668972739712/t8589944754(8589944754) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827182 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1787827170.650454:0:95229:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78a7ad00 x1874668972739712/t8589944754(8589944754) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827182 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827170.650466:0:95229:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827170.650477:0:95229:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589935618/512 (0 0) 00000080:00200000:1.0:1787827170.650482:0:95229:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827170.650495:0:95229:0:(namei.c:1628:ll_create_it()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248), intent=open|creat 00000080:00200000:1.0:1787827170.650517:0:95229:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000402:0x200:0x0](ffff9c1a813ff448) 00000080:00200000:1.0:1787827170.650534:0:95229:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000402:0x200:0x0], updating i_size 0 00000080:00200000:1.0:1787827170.650563:0:95229:0:(file.c:6854:ll_layout_conf()) [0x200000402:0x200:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1787827170.650567:0:95229:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200000402:0x200:0x0](ffff9c1a813ff448) 00000080:00010000:1.0:1787827170.650570:0:95229:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x200:0x0](ffff9c1a813ff448) for lock 0x531a24091f988866 00000080:00010000:1.0:1787827170.650578:0:95229:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x200:0x0](ffff9c1a813ff448) 00000080:00200008:1.0:1787827170.650594:0:95229:0:(file.c:1090:ll_file_open()) START file f41i.sanityn:[0x200000402:0x200:0x0](ffff9c1a813ff448), flags 100300 00000002:00100000:1.0:1787827170.650618:0:95229:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9c1a78a7ad00 x1874668972739712/t8589944754(8589944754) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827182 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1787827170.650657:0:95229:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988866 from it ffff9c1a529285a0 00010000:00010000:1.0:1787827170.650664:0:95229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827170.650673:0:95229:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827170.650692:0:95229:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000402:0x200:0x0](ffff9c1a813ff448), inode mode 81a4 mask 40 00000080:00200008:1.0:1787827170.650721:0:95229:0:(file.c:479:ll_file_release()) START file f41i.sanityn:[0x200000402:0x200:0x0](ffff9c1a813ff448), flags 100000 00000002:00100000:1.0:1787827170.650752:0:95229:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9c1a78a7ad00 x1874668972739712/t8589944754(8589944754) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827182 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787827170.650766:0:95229:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787827170.650775:0:95229:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78a79680 x1874668972741120/t0(0) o35->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1787827170.650804:0:95229:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79680 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95229:1874668972741120:192.168.203.148@tcp:35:multiop.0 00000100:00100000:1.0:1787827170.650875:0:95229:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43695300 going to sleep for 11 seconds 00000100:00100000:1.0:1787827170.651715:0:95229:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9c1a46db5000: committing for last_committed 8589944753 gen 1 00000100:00100000:1.0:1787827170.651735:0:95229:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff9c1a78a7ad00 x1874668972739712/t8589944754(8589944754) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827182 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1787827170.651744:0:95229:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79680 pname:cluuid:pid:xid:nid:opc:job multiop:7127393f-350f-49ff-88de-bcfd4bafe057:95229:1874668972741120:192.168.203.148@tcp:35:multiop.0 00000002:00100000:1.0:1787827170.651758:0:95229:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9c1a78a7ad00 x1874668972739712/t8589944754(8589944754) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 656/608 e 0 to 0 dl 1787827182 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00010000:00010000:3.0:1787827170.743516:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827170.999566:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2411:1874668972741248:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827170.999567:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972741376:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827170.999667:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972741632:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827170.999668:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972741504:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827170.999711:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972741760:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827171.001226:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2411:1874668972741248:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827171.001234:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972741376:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827171.001244:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972741504:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827171.001252:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972741632:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827171.001270:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972741760:192.168.203.148@tcp:400:kworker.0 00000080:00200000:0.0:1787827171.013082:0:95268:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827171.013108:0:95268:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827171.013161:0:95268:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 95268 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827171.013171:0:95268:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95268 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827171.013180:0:95268:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 95268 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1787827171.013203:0:95268:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95268:1874668972741888:192.168.203.148@tcp:101:rm.0 00000100:00100000:0.0:1787827171.013270:0:95268:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146600 going to sleep for 12 seconds 00000100:00100000:0.0:1787827171.014687:0:95268:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95268:1874668972741888:192.168.203.148@tcp:101:rm.0 00010000:00010000:0.0:1787827171.014703:0:95268:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827171.014713:0:95268:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827171.014725:0:95268:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827171.014734:0:95268:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827171.014744:0:95268:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827171.014752:0:95268:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827171.014759:0:95268:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1787827171.014771:0:95268:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9c1a78863480 x1874668972741888/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 576/688 e 0 to 0 dl 1787827188 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1787827171.014783:0:95268:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827171.014794:0:95268:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a466e0800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1787827171.014799:0:95268:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1787827171.014821:0:95268:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1787827171.014828:0:95268:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98887b 00000080:00010000:0.0:1787827171.014833:0:95268:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827171.014837:0:95268:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98887b from it ffffbe1909c0bb58 00010000:00010000:0.0:1787827171.014840:0:95268:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1787827171.014861:0:95268:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827171.014881:0:95268:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827171.014891:0:95268:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=8 00000080:00200008:0.0:1787827171.014899:0:95268:0:(file.c:6313:ll_getattr_dentry()) START file f41i.sanityn:[0x200000402:0x200:0x0](ffff9c1a813ff448), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1787827171.014904:0:95268:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x200:0x0](ffff9c1a813ff448),name=f41i.sanityn 00000002:00010000:0.0:1787827171.014918:0:95268:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x200:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827171.014927:0:95268:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827171.014938:0:95268:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827171.014945:0:95268:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827171.014950:0:95268:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x200:0x0](ffff9c1a813ff448) 00000080:00010000:0.0:1787827171.014952:0:95268:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000402:0x200:0x0](ffff9c1a813ff448) for lock 0x531a24091f988866 00000080:00010000:0.0:1787827171.014956:0:95268:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f988866 from it ffffbe1909c0bcb8 00010000:00010000:0.0:1787827171.014960:0:95268:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827171.014972:0:95268:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827171.015006:0:95268:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827171.015014:0:95268:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827171.015020:0:95268:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827171.015028:0:95268:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827171.015034:0:95268:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827171.015038:0:95268:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827171.015040:0:95268:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98887b 00000080:00010000:0.0:1787827171.015044:0:95268:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98887b from it ffffbe1909c0bc10 00010000:00010000:0.0:1787827171.015047:0:95268:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827171.015054:0:95268:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827171.015062:0:95268:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 1 00000080:00200000:0.0:1787827171.015069:0:95268:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f41i.sanityn, flags=0 00000080:00200000:0.0:1787827171.015072:0:95268:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248),name=/ 00000002:00010000:0.0:1787827171.015077:0:95268:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1787827171.015081:0:95268:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827171.015087:0:95268:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827171.015091:0:95268:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1787827171.015094:0:95268:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) 00000080:00010000:0.0:1787827171.015096:0:95268:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9c1a50d2b248) for lock 0x531a24091f98887b 00000080:00010000:0.0:1787827171.015099:0:95268:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x531a24091f98887b from it ffffbe1909c0bd68 00010000:00010000:0.0:1787827171.015102:0:95268:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827171.015108:0:95268:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827171.015115:0:95268:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d2b248), inode mode 41ff mask 3 00000080:00200000:0.0:1787827171.015120:0:95268:0:(namei.c:2418:ll_unlink()) VFS Op:name=f41i.sanityn, dir=[0x200000007:0x1:0x0](ffff9c1a50d2b248) 00010000:00010000:0.0:1787827171.015142:0:95268:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827171.015163:0:95268:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827171.015173:0:95268:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x200:0x0]: losing layout lock 00010000:00010000:0.0:1787827171.015194:0:95268:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827171.015201:0:95268:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827171.015210:0:95268:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1787827171.015211:0:95268:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a46ef6000/0x531a24091f988866 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x200:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bb337f expref: -99 pid: 95229 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827171.015222:0:95268:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a46db5000 lock: ffff9c1a59013c00/0x531a24091f98887b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bb33b0 expref: -99 pid: 95268 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1787827171.015233:0:95268:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9c1a46db5000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1787827171.015238:0:95268:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9c1a78863480 x1874668972742144/t0(0) o36->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827171.015254:0:95268:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95268:1874668972742144:192.168.203.148@tcp:36:rm.0 00000100:00100000:0.0:1787827171.015300:0:95268:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a7d146600 going to sleep for 12 seconds 00000100:00100000:0.0:1787827171.016939:0:95268:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863480 pname:cluuid:pid:xid:nid:opc:job rm:7127393f-350f-49ff-88de-bcfd4bafe057:95268:1874668972742144:192.168.203.148@tcp:36:rm.0 00000080:00200000:0.0:1787827171.016972:0:95268:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x200:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827171.016979:0:95268:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x200:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1787827171.016984:0:95268:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000402:0x200:0x0](ffff9c1a813ff448) 00000080:00200000:3.0:1787827171.662023:0:95335:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9c1a50d900c8),name=/ 00000002:00010000:3.0:1787827171.662048:0:95335:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1787827171.662060:0:95335:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9c1a46470400 type 13 mode 30 res 8589934599/1 (0 0) 00010000:00010000:3.0:1787827171.662106:0:95335:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e20000/0x531a24091f988890 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 95335 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827171.662116:0:95335:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e20000/0x531a24091f988890 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 95335 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1787827171.662126:0:95335:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e20000/0x531a24091f988890 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 95335 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1787827171.662151:0:95335:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d680 pname:cluuid:pid:xid:nid:opc:job multiop:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:95335:1874668972742656:192.168.203.148@tcp:101:multiop.0 00000100:00100000:3.0:1787827171.662218:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 12 seconds 00010000:00010000:0.0:1787827171.767455:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827172.791562:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827173.815520:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827174.839520:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.863525:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827176.119695:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7127393f-350f-49ff-88de-bcfd4bafe057:2413:1874668972742912:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.119786:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972743168:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.119810:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972743424:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.119834:0:2413:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:3.0:1787827176.119838:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2413:1874668972742784:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.119861:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7127393f-350f-49ff-88de-bcfd4bafe057:2413:1874668972743040:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.119887:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972743296:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.119906:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972743552:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.122172:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7127393f-350f-49ff-88de-bcfd4bafe057:2413:1874668972743040:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.122189:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972743296:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.122201:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972743552:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.122211:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7127393f-350f-49ff-88de-bcfd4bafe057:2413:1874668972742912:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.122225:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972743168:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.122236:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972743424:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827176.122269:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2413:1874668972742784:192.168.203.148@tcp:400:kworker.0 00010000:00010000:0.0:1787827176.887620:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827177.911561:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827178.935529:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787827179.959522:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827180.983487:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827181.239569:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2413:1874668972743680:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827181.239643:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7127393f-350f-49ff-88de-bcfd4bafe057:2413:1874668972743936:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827181.239665:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972744192:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827181.239688:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972744448:192.168.203.148@tcp:400:kworker.0 00010000:00010000:1.0:1787827182.007517:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827183.031502:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827183.799399:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 1 seconds 00010000:00010000:0.0:1787827184.055523:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827184.823462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 1 seconds 00010000:00010000:3.0:1787827185.079472:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827185.847488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 1 seconds 00010000:00010000:0.0:1787827186.103498:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827186.359622:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7127393f-350f-49ff-88de-bcfd4bafe057:2413:1874668972744704:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827186.359722:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972744960:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827186.359765:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972745216:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827186.359797:0:2413:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1787827186.359803:0:2413:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2413:1874668972745344:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827186.871469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 1 seconds 00010000:00010000:3.0:1787827187.127504:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827187.895501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 1 seconds 00010000:00010000:1.0:1787827188.151514:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1787827188.919460:0:95335:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827171/real 1787827171] req@ffff9c1a7895d680 x1874668972742656/t0(0) o101->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 576/1152 e 0 to 1 dl 1787827188 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1787827188.919496:0:95335:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1787827188.919501:0:95335:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9c1a49bbf800: Connection to lustre-MDT0000 (at 192.168.203.148@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1787827188.919524:0:95335:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00100000:3.0:1787827188.919546:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787827188.919626:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827188.919630:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827188.919632:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827188.919636:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 2715 00000100:00080000:0.0:1787827188.919647:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827188.919654:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827188.919663:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972745472:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827188.919674:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972745856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827188.919714:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972745856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827188.919718:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972745600:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827188.919752:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972745728:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827188.919761:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972745984:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827188.919793:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972746240:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827188.919798:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972746112:192.168.203.148@tcp:400:kworker.0 00000100:00080000:1.0:1787827188.931720:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827188.931726:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827188.931729:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827188.931736:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 2715 00000100:00080000:1.0:1787827188.931747:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827188.931756:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827188.931781:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972746368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827188.931838:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972746368:192.168.203.148@tcp:38:kworker.0 00010000:00010000:1.0:1787827189.175458:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827189.943454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827190.199496:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827190.967461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827191.223538:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827191.991467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827192.247538:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827193.015480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827193.271557:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827194.039443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787827194.039488:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827194.039493:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827194.039505:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827194.039506:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972746496:192.168.203.148@tcp:400:kworker.0 00000100:00080000:0.0:1787827194.039511:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1787827194.039645:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972746624:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827194.039646:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972746752:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827194.039710:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972746880:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827194.039715:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972747008:192.168.203.148@tcp:400:kworker.0 00010000:00010000:3.0:1787827194.295581:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827195.063462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827195.319518:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827196.087428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827196.343523:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827197.111456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827197.367508:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1787827197.623443:0:2414:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827181/real 1787827181] req@ffff9c1a7895e1c0 x1874668972744320/t0(0) o400->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827197.632471:0:2414:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a7895e1c0 x1874668972744320/t0(0) o400->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827197 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827197.632480:0:2414:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1787827197.632484:0:2414:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff9c1a49bbf800: Connection to lustre-OST0000 (at 192.168.203.148@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:0.0:1787827197.636080:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972746496:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827197.636105:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2412:1874668972745472:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827197.636206:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972747136:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827197.644734:0:2414:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00100000:3.0:1787827197.644815:0:2414:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2414:1874668972747392:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827197.644832:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972747520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827197.644920:0:2414:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2414:1874668972744576:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827197.644928:0:2414:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2414:1874668972744320:192.168.203.148@tcp:400:kworker.0 00000100:00080000:0.0:1787827197.644962:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827197.644968:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827197.644972:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827197.644976:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827197.644979:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827197.644983:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827197.644988:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827197.644994:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827197.644997:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827197.645005:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 2909 00000100:00080000:0.0:1787827197.645017:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827197.645026:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827197.645046:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972747776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827197.645096:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972747776:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827198.135422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827198.391508:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827198.647458:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827188/real 1787827188] req@ffff9c1a78a7b0c0 x1874668972746368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827198.647493:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972746368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827198 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827198.647507:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827188/real 1787827188] req@ffff9c1a78863840 x1874668972745856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827198.647520:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972745856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827198 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827198.647550:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827198.647643:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827198.647647:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972746368:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827198.647652:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827198.647657:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827198.647660:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972745856:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827198.647677:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827198.647685:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827198.647688:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000400:3.0:1787827198.647690:0:2414:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827181/real 1787827181] req@ffff9c1a7895cb40 x1874668972743808/t0(0) o400->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827198.647697:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827198.647703:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827198.647708:0:2414:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a7895cb40 x1874668972743808/t0(0) o400->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827198 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827198.647714:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3644 00000100:00080000:0.0:1787827198.647726:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787827198.647733:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787827198.647738:0:2414:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a450ef000 already not connected (conn 6, was 4): CONNECTING 00000008:00100000:0.0:1787827198.647745:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000400:3.0:1787827198.647747:0:2414:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827181/real 1787827181] req@ffff9c1a7895f840 x1874668972744064/t0(0) o400->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827198.647764:0:2414:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a7895f840 x1874668972744064/t0(0) o400->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827198 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827198.647772:0:2414:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a4a1eb000 already not connected (conn 5, was 4): DISCONN 00000100:00100000:0.0:1787827198.647778:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972747904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827198.647788:0:2414:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7127393f-350f-49ff-88de-bcfd4bafe057:2414:1874668972743808:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827198.647797:0:2414:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2414:1874668972744064:192.168.203.148@tcp:400:kworker.0 00000100:00080000:0.0:1787827198.647839:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827198.647845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972747904:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827198.647846:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827198.647850:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827198.647861:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3644 00000100:00080000:0.0:1787827198.647866:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1787827198.647870:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827198.647887:0:2411:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972748032:192.168.203.148@tcp:400:kworker.0 00000008:00100000:1.0:1787827198.647955:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827198.647975:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972748160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827198.648018:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827198.648024:0:2412:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972748288:192.168.203.148@tcp:400:kworker.0 00000100:00080000:1.0:1787827198.648025:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827198.648030:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827198.648035:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827198.648057:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972748160:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827198.658954:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827198.658961:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827198.658965:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827198.658970:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827198.658973:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827198.658980:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 259 00000100:00080000:3.0:1787827198.658992:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827198.659003:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827198.659029:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972748416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827198.659060:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827198.659065:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827198.659068:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827198.659073:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 318 00000100:00080000:3.0:1787827198.659079:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827198.659085:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827198.659096:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972748416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827198.659102:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972748544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827198.659116:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827198.659120:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827198.659123:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827198.659126:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1787827198.659164:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972748544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827199.159443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827199.415522:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827200.183415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827200.439470:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827201.207448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827201.463527:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827202.231411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827202.487515:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827202.679476:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827202.679484:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827202.679488:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827202.679491:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827202.679494:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827202.679497:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827202.679502:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827202.679505:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827202.679508:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827202.679510:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827202.679513:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827202.679516:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827202.679518:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827202.679520:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827203.255447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827203.511534:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827203.703480:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827203.703491:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827203.703495:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827203.703500:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827203.703503:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827203.703508:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827203.703513:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827203.703517:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827203.703521:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827203.703525:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827203.703529:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827203.703533:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827203.703536:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827203.703540:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:3.0:1787827203.767467:0:2414:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827186/real 1787827186] req@ffff9c1a7895fc00 x1874668972744832/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827203.767508:0:2414:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a7895fc00 x1874668972744832/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827203 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827203.767518:0:2414:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a450e8800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:3.0:1787827203.767529:0:2414:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827186/real 1787827186] req@ffff9c1a7895d2c0 x1874668972745088/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827203.767623:0:2414:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a7895d2c0 x1874668972745088/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827203 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827203.767639:0:2414:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a4a1ee000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:3.0:1787827203.767652:0:2414:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7127393f-350f-49ff-88de-bcfd4bafe057:2414:1874668972744832:192.168.203.148@tcp:400:kworker.0 00000100:00100000:3.0:1787827203.767662:0:2414:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2414:1874668972745088:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827204.279458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827204.535480:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1787827204.791442:0:2412:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827188/real 1787827188] req@ffff9c1a788612c0 x1874668972746112/t0(0) o400->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827204.791469:0:2412:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a788612c0 x1874668972746112/t0(0) o400->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827204 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827204.791477:0:2412:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9c1a4a1e9800 already not connected (conn 16, was 15): CONNECTING 00000100:00100000:1.0:1787827204.791492:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972746112:192.168.203.148@tcp:400:kworker.0 00000100:00000400:0.0:1787827204.791534:0:2411:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827188/real 1787827188] req@ffff9c1a78863c00 x1874668972746240/t0(0) o400->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827204.791551:0:2411:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a78863c00 x1874668972746240/t0(0) o400->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827204 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827204.791558:0:2411:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9c1a4a1ea800 already not connected (conn 12, was 11): CONNECTING 00000100:00100000:0.0:1787827204.791567:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972746240:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827205.303449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827205.559501:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1787827205.815476:0:2412:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827188/real 1787827188] req@ffff9c1a78862d00 x1874668972745728/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1787827205.815499:0:2411:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827188/real 1787827188] req@ffff9c1a788630c0 x1874668972745600/t0(0) o400->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827205.815520:0:2412:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a78862d00 x1874668972745728/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827205 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827205.815526:0:2411:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a788630c0 x1874668972745600/t0(0) o400->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827205 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827205.815539:0:2412:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a450e8800 already not connected (conn 2, was 1): CONNECTING 00000100:00080000:0.0:1787827205.815541:0:2411:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a450ef000 already not connected (conn 6, was 4): CONNECTING 00000100:00000400:0.0:1787827205.815549:0:2411:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827188/real 1787827188] req@ffff9c1a78861680 x1874668972745984/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827205.815564:0:2411:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a78861680 x1874668972745984/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827205 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827205.815565:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7127393f-350f-49ff-88de-bcfd4bafe057:2412:1874668972745728:192.168.203.148@tcp:400:kworker.0 00000100:00080000:0.0:1787827205.815578:0:2411:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a4a1ee000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:0.0:1787827205.815591:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972745600:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827205.815600:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972745984:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827206.327500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827206.583513:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827207.351451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827207.607534:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827207.800483:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827197/real 1787827197] req@ffff9c1a78862580 x1874668972747776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827207.800530:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972747776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827207.800545:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827197/real 1787827197] req@ffff9c1a49d8c780 x1874668972747520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827207.800562:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972747520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827207.800570:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827197/real 1787827197] req@ffff9c1a49d8cf00 x1874668972747136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827207.800579:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972747136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827207.800599:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827207.800624:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827207.800627:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972747776:192.168.203.148@tcp:8:kworker.0 00000100:00080000:0.0:1787827207.800632:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827207.800636:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827207.800639:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972747520:192.168.203.148@tcp:8:kworker.0 00000100:00080000:0.0:1787827207.800646:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827207.800652:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827207.800656:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972747136:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827207.800802:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972748672:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827207.800957:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972748800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827207.801000:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972748928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827208.375458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827208.631547:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827208.823623:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827198/real 1787827198] req@ffff9c1a7895cb40 x1874668972748544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827208.823669:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972748544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827208 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827208.823685:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827198/real 1787827198] req@ffff9c1a7895f840 x1874668972748416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827208.823701:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972748416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827208 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827208.823732:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827208.823768:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827208.823773:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972748544:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827208.823780:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827208.823802:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827208.823810:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972748416:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827208.824553:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827208.824563:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827208.824566:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827208.824569:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827208.824572:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827208.824576:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827208.824580:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827208.824591:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3654 00000100:00080000:3.0:1787827208.824606:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1787827208.824613:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827208.824626:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827208.824656:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972749056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827208.824684:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827208.824688:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827208.824691:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827208.824694:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827208.824696:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827208.824701:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3654 00000100:00080000:3.0:1787827208.824706:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1787827208.824713:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827208.824721:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827208.824735:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972749184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827208.824741:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972749056:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827208.824746:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827208.824751:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827208.824753:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827208.824756:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827208.824860:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972749184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827209.399457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827209.655538:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827210.423463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827210.679544:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1787827210.935460:0:2411:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827194/real 1787827194] req@ffff9c1a78860f00 x1874668972746880/t0(0) o400->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1787827210.935541:0:2412:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827194/real 1787827194] req@ffff9c1a788621c0 x1874668972747008/t0(0) o400->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827210.935570:0:2412:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a788621c0 x1874668972747008/t0(0) o400->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827210 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827210.935581:0:2412:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9c1a4a1ea800 already not connected (conn 13, was 11): CONNECTING 00000100:00100000:0.0:1787827210.935593:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972747008:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827210.945456:0:2411:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a78860f00 x1874668972746880/t0(0) o400->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827210 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827210.945465:0:2411:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9c1a4a1e9800 already not connected (conn 17, was 15): CONNECTING 00000100:00100000:1.0:1787827210.945477:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2411:1874668972746880:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827211.447530:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827211.703550:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1787827211.959461:0:2412:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827194/real 1787827194] req@ffff9c1a788603c0 x1874668972746752/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1787827211.959461:0:2411:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827194/real 1787827194] req@ffff9c1a78861a40 x1874668972746624/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827211.959502:0:2412:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a788603c0 x1874668972746752/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827211 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827211.959511:0:2411:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a78861a40 x1874668972746624/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827211 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827211.959513:0:2412:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a4a1ee000 already not connected (conn 3, was 1): CONNECTING 00000100:00080000:1.0:1787827211.959524:0:2411:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a450e8800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1787827211.959528:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972746752:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827211.959536:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972746624:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827212.471445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827212.727531:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827212.855466:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827212.855476:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827212.855480:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827212.855484:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827212.855486:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827212.855490:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827212.855494:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827212.855498:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827212.855500:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827212.855504:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827212.855507:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827212.855509:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827212.855512:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827212.855515:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827213.495410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827213.751559:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827213.879463:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.879471:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827213.879474:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.879476:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827213.879478:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.879480:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827213.879483:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.879486:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827213.879487:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.879490:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827213.879492:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.879494:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827213.879496:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827213.879499:0:60751:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827213.943424:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827198/real 1787827198] req@ffff9c1a78a7bc00 x1874668972748160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827213 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827213.943461:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972748160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827213 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827213.943476:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827198/real 1787827198] req@ffff9c1a78860b40 x1874668972747904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827213 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827213.943490:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972747904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827213 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827213.943508:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827213.943526:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827213.943529:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972748160:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827213.943534:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827213.943537:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827213.943539:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972747904:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827213.943562:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827213.943567:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827213.943570:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827213.943572:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827213.943575:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827213.943581:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3654 00000100:00080000:0.0:1787827213.943590:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1787827213.943595:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827213.943606:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827213.943630:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788603c0 x1874668972749312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827213.943648:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827213.943650:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827213.943653:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827213.943655:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827213.943657:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827213.943659:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3654 00000100:00080000:0.0:1787827213.943662:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1787827213.943665:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827213.943669:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827213.943676:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972749312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827213.943679:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972749440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827213.943690:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827213.943692:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827213.943694:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827213.943696:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827213.943697:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827213.943700:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827213.943765:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972749440:192.168.203.148@tcp:38:kworker.0 00000100:00000400:3.0:1787827214.007465:0:2413:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827197/real 1787827197] req@ffff9c1a49d8da40 x1874668972747264/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827214.007510:0:2413:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a49d8da40 x1874668972747264/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827214 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827214.007540:0:2413:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a450e8800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:3.0:1787827214.007555:0:2413:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7127393f-350f-49ff-88de-bcfd4bafe057:2413:1874668972747264:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827214.519446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827214.775531:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1787827215.031449:0:2411:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827198/real 1787827198] req@ffff9c1a78862940 x1874668972748032/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.031476:0:2411:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a78862940 x1874668972748032/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827215 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827215.031484:0:2411:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a450e8800 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:0.0:1787827215.031498:0:2412:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827198/real 1787827198] req@ffff9c1a78a79e00 x1874668972748288/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.031501:0:2411:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7127393f-350f-49ff-88de-bcfd4bafe057:2411:1874668972748032:192.168.203.148@tcp:400:kworker.0 00000100:00100000:0.0:1787827215.031524:0:2412:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9c1a78a79e00 x1874668972748288/t0(0) o400->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827215 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827215.031531:0:2412:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9c1a4a1ee000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1787827215.031541:0:2412:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2412:1874668972748288:192.168.203.148@tcp:400:kworker.0 00000100:00100000:1.0:1787827215.543427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827215.799502:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827216.567440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827216.823538:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827217.591431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827217.847508:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827218.615455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827218.871539:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827219.127442:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.127450:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827219.127453:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.127455:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827219.127457:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.127459:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827219.127461:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.127463:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827219.127465:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.127467:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827219.127468:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.127470:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1787827219.127472:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827219.127474:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827219.639413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827219.895589:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827220.663462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827220.919619:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827221.687497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827221.943495:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827222.135494:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827207/real 1787827207] req@ffff9c1a49d8de00 x1874668972748800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827222.135532:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972748800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827222 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827222.135541:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827207/real 1787827207] req@ffff9c1a49d8f480 x1874668972748928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827222.135551:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f480 x1874668972748928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827222 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827222.135557:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827207/real 1787827207] req@ffff9c1a49d8d680 x1874668972748672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827222.135566:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972748672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827222 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827222.135591:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827222.135626:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827222.135631:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972748800:192.168.203.148@tcp:8:kworker.0 00000100:00080000:1.0:1787827222.135638:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827222.135644:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827222.135647:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972748928:192.168.203.148@tcp:8:kworker.0 00000100:00080000:1.0:1787827222.135650:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827222.135654:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827222.135656:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972748672:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827222.135721:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827222.135726:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827222.135729:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827222.135735:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3663 00000100:00080000:3.0:1787827222.135747:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787827222.135754:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827222.135793:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972749568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827222.135813:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827222.135817:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827222.135819:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827222.135822:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827222.135825:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827222.135828:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827222.135830:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827222.135832:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827222.135834:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827222.135836:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827222.135837:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827222.135841:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3663 00000100:00080000:3.0:1787827222.135843:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787827222.135849:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827222.135851:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972749568:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787827222.135861:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827222.135871:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972749696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827222.135878:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827222.135881:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827222.135882:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827222.135885:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3663 00000100:00080000:3.0:1787827222.135887:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787827222.135890:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827222.135893:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827222.135900:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972749824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827222.135948:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972749696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827222.135969:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972749824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827222.711473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827222.967577:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827223.159472:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827208/real 1787827208] req@ffff9c1a7895fc00 x1874668972749184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827223.159509:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972749184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827223.159522:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827208/real 1787827208] req@ffff9c1a7895d2c0 x1874668972749056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827223.159544:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972749056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827223 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827223.159576:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827223.159608:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827223.159615:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972749184:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827223.159630:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827223.159648:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827223.159650:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972749056:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827223.159656:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827223.159665:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827223.159669:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827223.159672:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827223.159676:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827223.159679:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827223.159684:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827223.159693:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3664 00000100:00080000:3.0:1787827223.159707:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787827223.159714:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827223.159728:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827223.159766:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972749952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827223.159793:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827223.159796:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827223.159799:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827223.159801:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827223.159802:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827223.159805:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3664 00000100:00080000:3.0:1787827223.159807:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1787827223.159810:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827223.159813:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827223.159821:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972750080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827223.159828:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827223.159831:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827223.159833:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827223.159835:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827223.159876:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972749952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827223.159942:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972750080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827223.735461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827223.991533:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827224.759491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1787827224.887720:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.148:988 00000400:00000100:0.0:1787827224.887771:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827224.887787:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827224.887835:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827224.887846:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827224.887855:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827224.887865:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787827224.887986:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827223/real 1787827224] req@ffff9c1a7895d2c0 x1874668972749952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827224.888023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972749952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827224.888036:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827224.888042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827224.888047:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972749952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827224.888056:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827223/real 1787827224] req@ffff9c1a7895fc00 x1874668972750080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827224.888088:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972750080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827224.888096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827224.888100:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827224.888104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972750080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827224.888111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827222/real 1787827224] req@ffff9c1a7895f0c0 x1874668972749696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827224.888122:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972749696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827224.888130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827224.888133:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827224.888136:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972749696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827224.888143:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827222/real 1787827224] req@ffff9c1a7895da40 x1874668972749824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827224.888172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972749824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827224.888180:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827224.888183:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827224.888187:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972749824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827224.888194:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827222/real 1787827224] req@ffff9c1a7895e1c0 x1874668972749568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827224.888205:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972749568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827224.888214:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827224.888218:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827224.888221:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972749568:192.168.203.148@tcp:250:kworker.0 00010000:00010000:1.0:1787827225.015527:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827225.783421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827226.039540:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827226.807472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827227.063543:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827227.191413:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827227.191422:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827227.191425:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827227.191433:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3677 00000100:00080000:3.0:1787827227.191443:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1787827227.191449:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827227.191484:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972750208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827227.191512:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827227.191516:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827227.191520:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827227.191522:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827227.191524:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827227.191528:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3678 00000100:00080000:3.0:1787827227.191533:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1787827227.191539:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827227.191550:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827227.191561:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972750208:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827227.191566:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972750336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827227.191580:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827227.191584:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1787827227.191587:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827227.191592:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827227.191593:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827227.191596:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3678 00000100:00080000:3.0:1787827227.191600:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1787827227.191603:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827227.191607:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827227.191622:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972750464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827227.191632:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827227.191636:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827227.191638:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827227.191641:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3677 00000100:00080000:3.0:1787827227.191646:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1787827227.191650:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827227.191656:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827227.191671:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972750336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827227.191672:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972750592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827227.191689:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827227.191693:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827227.191695:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827227.191699:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3677 00000100:00080000:3.0:1787827227.191704:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 15s 00000100:00100000:1.0:1787827227.191710:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972750464:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827227.191711:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827227.191722:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827227.191739:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972750720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827227.191741:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972750592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827227.191771:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972750720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827227.831448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1787827227.959671:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.148:988 00000400:00000100:3.0:1787827227.959721:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827227.959735:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827227.959765:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827227.959774:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827227.959782:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827227.959791:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787827227.959824:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827227/real 1787827227] req@ffff9c1a7895c780 x1874668972750720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827227.959865:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972750720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827227.959876:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827227.959882:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827227.959886:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972750720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827227.959894:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827227/real 1787827227] req@ffff9c1a7895de00 x1874668972750592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827227.959912:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972750592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827227.959919:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827227.959922:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827227.959925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972750592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827227.959932:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827227/real 1787827227] req@ffff9c1a7895cb40 x1874668972750336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827227.959944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972750336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827227.959951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827227.959955:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827227.959958:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972750336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827227.959966:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827227/real 1787827227] req@ffff9c1a7895e580 x1874668972750464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827227.959979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972750464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827227.959987:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827227.959990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827227.959993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972750464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827227.960000:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827227/real 1787827227] req@ffff9c1a7895f840 x1874668972750208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827227.960013:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972750208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827227.960021:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827227.960024:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827227.960028:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972750208:192.168.203.148@tcp:250:kworker.0 00010000:00010000:3.0:1787827228.087511:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827228.855454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827229.111553:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827229.879497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827230.135583:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827230.903444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827231.159528:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827231.927465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827231.927469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827232.183466:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827232.439502:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827232.439512:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827232.439517:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827232.439529:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3682 00000100:00080000:1.0:1787827232.439541:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827232.439548:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827232.439593:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972750848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827232.439629:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827232.439634:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827232.439636:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827232.439640:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827232.439641:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827232.439646:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3682 00000100:00080000:1.0:1787827232.439650:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827232.439655:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827232.439665:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827232.439682:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972750976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827232.439688:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972750848:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827232.439711:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827232.439716:0:40547:0:(pinger.c:247:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827232.439720:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827232.439723:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827232.439726:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827232.439731:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3682 00000100:00080000:1.0:1787827232.439745:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827232.439751:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827232.439756:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827232.439771:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972751104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827232.439783:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827232.439787:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827232.439789:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827232.439795:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3682 00000100:00080000:1.0:1787827232.439799:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827232.439801:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972750976:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827232.439807:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827232.439816:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827232.439832:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972751232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827232.439844:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972751104:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827232.439847:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827232.439853:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827232.439856:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827232.439862:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3682 00000100:00080000:1.0:1787827232.439867:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827232.439874:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827232.439880:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972751232:192.168.203.148@tcp:8:kworker.0 00000008:00100000:1.0:1787827232.439881:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827232.439909:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972751360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827232.439970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972751360:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827232.951454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827232.951458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827233.207501:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827233.463466:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827213/real 1787827213] req@ffff9c1a788621c0 x1874668972749440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827233.463501:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972749440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827233.463509:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827213/real 1787827213] req@ffff9c1a788603c0 x1874668972749312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827233.463522:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788603c0 x1874668972749312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827233 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827233.463540:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827233.463560:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827233.463563:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972749440:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827233.463567:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827233.463571:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827233.463573:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972749312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827233.463736:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972751488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827233.463797:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972751616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827233.975464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827233.975465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000400:00000100:3.0:1787827234.103651:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.148:988 00000400:00000100:3.0:1787827234.103696:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827234.103710:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827234.103742:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827234.103750:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827234.103772:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827234.103784:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787827234.103786:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827232/real 1787827234] req@ffff9c1a78a79e00 x1874668972750976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787827234.103798:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787827234.103821:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972750976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787827234.103829:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00080000:0.0:1787827234.103842:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827234.103848:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827234.103853:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972750976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827234.103860:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827232/real 1787827234] req@ffff9c1a78a7b0c0 x1874668972751104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827234.103881:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972751104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827234.103890:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827234.103894:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827234.103897:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972751104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827234.103904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827232/real 1787827234] req@ffff9c1a78a7bc00 x1874668972750848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827234.103916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972750848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827234.103924:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827234.103928:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827234.103931:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972750848:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827234.103961:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827233/real 1787827234] req@ffff9c1a49d8f0c0 x1874668972751616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827234.103974:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972751616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827234.103982:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827234.103985:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827234.103989:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972751616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827234.103995:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827233/real 1787827234] req@ffff9c1a49d8c000 x1874668972751488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827234.104007:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972751488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827234.104015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827234.104019:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827234.104022:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972751488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827234.104028:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827232/real 1787827234] req@ffff9c1a78a78780 x1874668972751360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827234.104041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972751360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827234.104048:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827234.104052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827234.104055:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972751360:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827234.104061:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827232/real 1787827234] req@ffff9c1a78a7a1c0 x1874668972751232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827234.104072:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972751232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827234.104093:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827234.104097:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827234.104101:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972751232:192.168.203.148@tcp:8:kworker.0 00010000:00010000:0.0:1787827234.231524:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827234.999446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827234.999446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827235.255511:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827236.023469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827236.023471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827236.279534:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827237.047445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827237.303525:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827237.495478:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827237.495487:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827237.495491:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827237.495500:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3687 00000100:00080000:1.0:1787827237.495510:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827237.495517:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827237.495549:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972751744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827237.495569:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827237.495572:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827237.495573:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827237.495576:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3688 00000100:00080000:1.0:1787827237.495578:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827237.495580:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827237.495588:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827237.495596:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972751872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827237.495603:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827237.495606:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827237.495607:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827237.495610:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3687 00000100:00080000:1.0:1787827237.495613:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827237.495615:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827237.495619:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827237.495623:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972751744:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827237.495630:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972752000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827237.495643:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827237.495647:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827237.495649:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827237.495653:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3688 00000100:00080000:1.0:1787827237.495657:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827237.495661:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827237.495666:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827237.495680:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78b40 x1874668972752128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827237.495691:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787827237.495693:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827237.495699:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827237.495702:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827237.495718:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3687 00000100:00080000:1.0:1787827237.495723:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827237.495728:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827237.495734:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827237.495740:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827237/real 1787827237] req@ffff9c1a78a783c0 x1874668972751744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827237.495758:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972752256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827237.495763:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972751744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827237.495797:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827237.495801:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827237.495802:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827237.495809:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827237.495809:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827237.495816:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972751744:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827237.495820:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3687 00000100:00100000:0.0:1787827237.495826:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972751872:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827237.495827:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827237.495836:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827237.495841:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827237.495853:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827237.495854:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972752384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827237.495863:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827237.495867:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827237/real 1787827237] req@ffff9c1a78a7a940 x1874668972751872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827237.495868:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827237.495871:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827237.495878:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3687 00000100:00080000:1.0:1787827237.495884:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827237.495886:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972751872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827237.495890:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827237.495896:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827237.495899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827237.495904:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827237.495907:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972752512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827237.495909:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972751872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827237.495937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972752000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827237.495950:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827237.495958:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827237/real 1787827237] req@ffff9c1a78a7b840 x1874668972752000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827237.495966:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972752000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827237.495971:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827237.495973:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827237.495975:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972752000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827237.495978:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972752128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827237.495988:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827237.495993:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827237/real 1787827237] req@ffff9c1a78a78b40 x1874668972752128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827237.496001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78b40 x1874668972752128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827237.496006:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827237.496007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827237.496009:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972752128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827237.496013:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972752256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827237.496022:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827237.496027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827237/real 1787827237] req@ffff9c1a78a78f00 x1874668972752256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827237.496034:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972752256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827237.496044:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827237.496045:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827237.496047:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972752256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827237.496050:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972752384:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827237.496079:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827237.496091:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827237/real 1787827237] req@ffff9c1a78a792c0 x1874668972752384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827237.496103:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972752384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827237.496111:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827237.496115:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827237.496118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972752384:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827237.496124:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972752512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827237.496143:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827237.496155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827237/real 1787827237] req@ffff9c1a78a7b480 x1874668972752512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827237.496167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972752512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827237.496175:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827237.496177:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827237.496179:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972752512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827238.071447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827238.327530:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827239.095465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827239.351547:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827240.119494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827240.375565:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827241.143453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827241.399547:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827242.167493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827242.423519:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827242.679599:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972752640:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827242.679693:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972752768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827242.679726:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972752896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827242.679770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972753024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827242.679798:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972753152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827242.679823:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972753280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827242.679849:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972753408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827243.191446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1787827243.319700:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.148:988 00000400:00000100:1.0:1787827243.319770:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827243.319788:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827243.319843:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827243.319854:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827243.319862:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827243.319870:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827243.319879:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827243.319888:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787827243.319936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827242/real 1787827243] req@ffff9c1a49d8e940 x1874668972753408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827243.319968:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e940 x1874668972753408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827243.319982:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827243.319989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827243.319994:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972753408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827243.320003:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827242/real 1787827243] req@ffff9c1a49d8cb40 x1874668972753280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827243.320037:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972753280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827243.320046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827243.320049:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827243.320053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972753280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827243.320060:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827242/real 1787827243] req@ffff9c1a49d8ed00 x1874668972753024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827243.320071:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972753024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827243.320078:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827243.320082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827243.320086:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972753024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827243.320093:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827242/real 1787827243] req@ffff9c1a49d8c3c0 x1874668972753152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827243.320105:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972753152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827243.320112:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827243.320116:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827243.320119:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972753152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827243.320126:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827242/real 1787827243] req@ffff9c1a49d8e580 x1874668972752768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827243.320138:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972752768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827243.320148:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827243.320152:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827243.320155:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972752768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827243.320162:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827242/real 1787827243] req@ffff9c1a49d8fc00 x1874668972752896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827243.320173:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972752896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827243.320182:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827243.320185:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827243.320188:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972752896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827243.320195:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827242/real 1787827243] req@ffff9c1a49d8e1c0 x1874668972752640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827243.320208:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972752640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827243.320216:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827243.320219:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827243.320222:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972752640:192.168.203.148@tcp:250:kworker.0 00010000:00010000:0.0:1787827243.447492:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827244.215416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827244.471539:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827245.239457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827245.495544:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827246.263475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827246.519585:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827247.287452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827247.543542:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827247.799463:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827247.799471:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827247.799475:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827247.799482:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3698 00000100:00080000:3.0:1787827247.799492:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827247.799498:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827247.799528:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972753536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827247.799549:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827247.799552:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827247.799553:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827247.799556:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3698 00000100:00080000:3.0:1787827247.799558:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827247.799561:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827247.799568:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827247.799578:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972753664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827247.799585:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827247.799588:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827247.799589:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827247.799591:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3698 00000100:00080000:3.0:1787827247.799593:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827247.799595:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827247.799598:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827247.799608:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972753792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827247.799609:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972753536:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827247.799625:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827247.799629:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827247.799631:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827247.799635:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3698 00000100:00080000:3.0:1787827247.799639:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827247.799643:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827247.799649:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827247.799662:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972753920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827247.799671:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827247.799673:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827247.799675:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827247.799682:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827247.799682:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3698 00000100:00080000:3.0:1787827247.799688:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827247.799691:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827247.799696:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827247.799706:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972754048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827247.799712:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827247.799714:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827247.799716:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827247.799718:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3698 00000100:00080000:3.0:1787827247.799721:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827247.799723:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827247.799728:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827247.799729:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827247/real 1787827247] req@ffff9c1a7895e940 x1874668972753536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827247.799739:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972754176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827247.799751:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827247.799753:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827247.799755:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827247.799759:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972753536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827247.799760:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3698 00000100:00080000:3.0:1787827247.799765:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827247.799771:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827247.799777:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827247.799785:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827247.799789:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972754304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827247.799794:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827247.799801:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972753536:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827247.799816:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972753664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827247.799842:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827247.799855:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827247/real 1787827247] req@ffff9c1a7895c3c0 x1874668972753664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827247.799869:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972753664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827247.799878:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827247.799881:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827247.799885:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972753664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827247.799907:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972753792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827247.799927:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827247.799938:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827247/real 1787827247] req@ffff9c1a7895cf00 x1874668972753792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827247.799952:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972753792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827247.799960:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827247.799963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827247.799967:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972753792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827247.799973:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972753920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827247.800007:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827247.800019:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827247/real 1787827247] req@ffff9c1a7895f480 x1874668972753920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827247.800031:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972753920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827247.800039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827247.800043:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827247.800046:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972753920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827247.800052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972754048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827247.800072:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827247.800084:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827247/real 1787827247] req@ffff9c1a7895c000 x1874668972754048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827247.800097:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972754048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827247.800110:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827247.800113:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827247.800116:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972754048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827247.800122:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972754176:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827247.800141:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827247.800151:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827247/real 1787827247] req@ffff9c1a7895f840 x1874668972754176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827247.800162:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972754176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827247.800170:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827247.800173:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827247.800176:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972754176:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827247.800181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972754304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827247.800196:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827247.800205:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827247/real 1787827247] req@ffff9c1a7895e580 x1874668972754304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827247.800216:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972754304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827247.800223:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827247.800226:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827247.800229:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972754304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827248.311439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827248.567510:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827249.335447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827249.591529:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827250.359465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827250.615586:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827251.383472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827251.639533:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827252.407425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827252.407483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827252.663519:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827252.919507:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827252.919516:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827252.919520:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827252.919530:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3703 00000100:00080000:1.0:1787827252.919541:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827252.919548:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827252.919588:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972754432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827252.919616:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827252.919620:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827252.919622:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827252.919628:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3703 00000100:00080000:1.0:1787827252.919631:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827252.919635:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827252.919642:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827252.919654:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972754560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827252.919666:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827252.919668:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827252.919670:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827252.919673:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3703 00000100:00080000:1.0:1787827252.919675:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827252.919677:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827252.919680:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827252.919688:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972754432:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827252.919688:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972754688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827252.919705:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827252.919707:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827252.919708:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827252.919710:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3703 00000100:00080000:1.0:1787827252.919712:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827252.919714:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827252.919717:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827252.919725:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972754816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827252.919732:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827252.919734:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827252.919735:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827252.919737:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3703 00000100:00080000:1.0:1787827252.919748:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827252.919751:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827252.919754:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827252.919762:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972754944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827252.919768:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827252.919770:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827252.919771:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827252.919777:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3703 00000100:00080000:1.0:1787827252.919779:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827252.919781:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827252.919785:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827252.919796:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972755072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827252.919800:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972754560:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827252.919815:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827252.919819:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827252.919821:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827252.919827:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3703 00000100:00080000:1.0:1787827252.919832:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827252.919837:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827252.919843:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827252.919852:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972754688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827252.919857:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78b40 x1874668972755200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827252.919888:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972754816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827252.919905:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972754944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827252.919926:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972755072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827252.919970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972755200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827253.431444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827253.431444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827253.687474:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827254.455418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827254.455423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827254.711507:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827255.479458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827255.479458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1787827255.607633:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.148:988 00000400:00000100:0.0:1787827255.607672:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827255.607687:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827255.607728:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827255.607738:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827255.607748:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827255.607759:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827255.607770:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787827255.607779:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827252/real 1787827255] req@ffff9c1a78a78000 x1874668972754688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827255.607790:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787827255.607819:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972754688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827255.607832:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827255.607838:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827255.607842:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972754688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827255.607854:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827252/real 1787827255] req@ffff9c1a78a7b480 x1874668972754816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827255.607867:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972754816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827255.607875:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827255.607879:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827255.607882:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972754816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827255.607889:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827252/real 1787827255] req@ffff9c1a78a792c0 x1874668972754944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827255.607900:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972754944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827255.607908:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827255.607912:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827255.607916:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972754944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827255.607922:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827252/real 1787827255] req@ffff9c1a78a78f00 x1874668972755072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827255.607947:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972755072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827255.607956:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827255.607959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827255.607963:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972755072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827255.607970:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827252/real 1787827255] req@ffff9c1a78a7a580 x1874668972754432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827255.607982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972754432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827255.607990:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827255.607993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827255.607996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972754432:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827255.608002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827252/real 1787827255] req@ffff9c1a78a79a40 x1874668972754560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827255.608013:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972754560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827255.608021:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827255.608024:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827255.608028:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972754560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827255.608055:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827252/real 1787827255] req@ffff9c1a78a78b40 x1874668972755200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827255.608069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78b40 x1874668972755200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827255.608076:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827255.608079:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827255.608083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972755200:192.168.203.148@tcp:8:kworker.0 00010000:00010000:0.0:1787827255.735489:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827256.503405:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827256.759449:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827257.527439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827257.783522:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827258.039473:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.039481:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827258.039484:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.039493:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3708 00000100:00080000:1.0:1787827258.039504:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.039511:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827258.039545:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78b40 x1874668972755328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827258.039567:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.039569:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827258.039571:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.039573:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3708 00000100:00080000:1.0:1787827258.039575:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.039578:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827258.039585:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827258.039594:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972755456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827258.039601:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.039603:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827258.039604:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.039607:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3708 00000100:00080000:1.0:1787827258.039609:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.039611:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827258.039614:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827258.039623:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972755584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827258.039625:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972755328:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827258.039635:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.039646:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827258.039647:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.039650:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3708 00000100:00080000:1.0:1787827258.039653:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.039656:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827258.039659:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827258.039669:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972755712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827258.039675:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.039677:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827258.039678:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.039680:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3708 00000100:00080000:1.0:1787827258.039684:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.039686:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827258.039688:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827258.039695:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972755840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827258.039704:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827258.039705:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.039710:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827258.039711:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.039714:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3708 00000100:00080000:1.0:1787827258.039716:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.039719:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827258.039723:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827258.039729:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972755968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827258.039735:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.039737:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827258.039738:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.039741:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3708 00000100:00080000:1.0:1787827258.039743:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.039746:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827258.039749:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827258.039755:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827258] req@ffff9c1a78a78b40 x1874668972755328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827258.039758:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972756096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827258.039802:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78b40 x1874668972755328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827258.039814:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827258.039818:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827258.039823:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972755328:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827258.039831:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972755456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827258.039862:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827258.039877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827258] req@ffff9c1a78a79a40 x1874668972755456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827258.039889:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972755456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827258.039898:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827258.039901:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827258.039905:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972755456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827258.039943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972755584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827258.039967:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827258.039978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827258] req@ffff9c1a78a7a580 x1874668972755584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827258.039990:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972755584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827258.039998:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827258.040001:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827258.040004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972755584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827258.040013:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972755712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827258.040031:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827258.040042:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827258] req@ffff9c1a78a78f00 x1874668972755712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827258.040053:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972755712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827258.040062:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827258.040065:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827258.040068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972755712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827258.040074:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972755840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827258.040094:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827258.040104:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827258] req@ffff9c1a78a792c0 x1874668972755840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827258.040123:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972755840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827258.040131:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827258.040134:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827258.040138:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972755840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827258.040144:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972755968:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827258.040163:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827258.040174:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827258] req@ffff9c1a78a7b480 x1874668972755968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827258.040185:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972755968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827258.040193:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827258.040196:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827258.040199:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972755968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827258.040205:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972756096:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827258.040222:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827258.040234:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827258] req@ffff9c1a78a78000 x1874668972756096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827258.040246:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972756096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827258.040259:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827258.040262:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827258.040265:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972756096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827258.551411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827258.807473:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827259.575438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827259.831525:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827260.599445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827260.855476:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827261.623440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827261.879513:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827262.647447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827262.903483:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827263.159652:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972756224:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787827263.159725:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827263.159776:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827263/real 1787827263] req@ffff9c1a49d8e1c0 x1874668972756224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827263.159800:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972756224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827263.159819:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827263.159827:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827263.159836:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972756224:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827263.159869:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972756352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827263.159940:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827263.159966:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827263/real 1787827263] req@ffff9c1a49d8fc00 x1874668972756352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827263.160001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972756352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827263.160019:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827263.160030:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827263.160040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972756352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827263.160056:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972756480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827263.160114:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827263.160126:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827263/real 1787827263] req@ffff9c1a49d8e580 x1874668972756480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827263.160140:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972756480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827263.160148:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827263.160151:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827263.160155:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972756480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827263.160162:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972756608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827263.160183:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827263.160194:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827263/real 1787827263] req@ffff9c1a49d8c3c0 x1874668972756608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827263.160207:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972756608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827263.160215:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827263.160218:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827263.160221:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972756608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827263.160227:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972756736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827263.160250:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827263.160261:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827263/real 1787827263] req@ffff9c1a49d8ed00 x1874668972756736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827263.160270:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972756736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827263.160275:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827263.160278:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827263.160280:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972756736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827263.160297:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972756864:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827263.160320:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827263.160331:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827263/real 1787827263] req@ffff9c1a49d8cb40 x1874668972756864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827263.160343:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972756864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827263.160367:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827263.160371:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827263.160404:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972756864:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827263.160413:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972756992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827263.160439:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827263.160452:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827263/real 1787827263] req@ffff9c1a49d8e940 x1874668972756992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827263.160469:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e940 x1874668972756992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827263.160491:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827263.160494:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827263.160496:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972756992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827263.671448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827263.927549:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827264.695454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827264.951557:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827265.719442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827265.975508:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827266.743458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827266.999528:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827267.767436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827268.023536:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827268.279479:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827268.279489:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827268.279492:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827268.279501:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3718 00000100:00080000:1.0:1787827268.279510:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827268.279516:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827268.279546:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972757120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827268.279568:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827268.279571:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827268.279572:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827268.279575:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3718 00000100:00080000:1.0:1787827268.279577:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827268.279579:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827268.279586:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827268.279595:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972757248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827268.279603:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827268.279605:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827268.279607:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827268.279611:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3718 00000100:00080000:1.0:1787827268.279613:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827268.279615:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827268.279619:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827268.279623:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972757120:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827268.279628:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972757376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827268.279637:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827268.279639:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827268.279640:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827268.279642:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3718 00000100:00080000:1.0:1787827268.279644:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827268.279646:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827268.279649:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827268.279657:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972757504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827268.279663:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827268.279666:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827268.279668:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827268.279672:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3718 00000100:00080000:1.0:1787827268.279684:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827268.279688:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787827268.279691:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:1.0:1787827268.279698:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827268.279715:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972757632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827268.279723:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827268.279726:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827268.279727:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827268.279731:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827268/real 1787827268] req@ffff9c1a78a7b840 x1874668972757120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827268.279733:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3718 00000100:00080000:1.0:1787827268.279740:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827268.279747:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827268.279751:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827268.279754:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972757120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827268.279762:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972757760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827268.279770:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827268.279771:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827268.279776:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827268.279780:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827268.279780:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827268.279788:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972757120:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827268.279789:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3718 00000100:00080000:1.0:1787827268.279794:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827268.279800:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972757248:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827268.279801:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827268.279808:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827268.279818:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972757888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827268.279837:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827268.279870:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827268/real 1787827268] req@ffff9c1a78a7a940 x1874668972757248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827268.279883:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972757248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827268.279892:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827268.279896:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827268.279899:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972757248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827268.279922:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972757376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827268.279942:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827268.279951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827268/real 1787827268] req@ffff9c1a78a783c0 x1874668972757376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827268.279959:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972757376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827268.279964:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827268.279966:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827268.279968:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972757376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827268.279972:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972757504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827268.279983:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827268.279989:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827268/real 1787827268] req@ffff9c1a78a7a1c0 x1874668972757504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827268.279996:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972757504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827268.280001:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827268.280003:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827268.280005:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972757504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827268.280009:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972757632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827268.280018:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827268.280024:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827268/real 1787827268] req@ffff9c1a78a78780 x1874668972757632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827268.280031:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972757632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827268.280036:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827268.280037:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827268.280039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972757632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827268.280042:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972757760:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827268.280052:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827268.280057:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827268/real 1787827268] req@ffff9c1a78a7bc00 x1874668972757760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827268.280065:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972757760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827268.280069:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827268.280071:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827268.280073:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972757760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827268.280076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972757888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827268.280086:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827268.280092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827268/real 1787827268] req@ffff9c1a78a7b0c0 x1874668972757888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827268.280099:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972757888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827268.280104:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827268.280105:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827268.280107:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972757888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827268.791526:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827269.047507:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827269.815467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827270.071553:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827270.839448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827271.095554:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827271.863424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827272.119506:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827272.887445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827273.143534:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827273.399444:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827273.399452:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827273.399456:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827273.399464:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3723 00000100:00080000:3.0:1787827273.399473:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827273.399479:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827273.399510:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972758016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827273.399534:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827273.399537:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827273.399539:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827273.399542:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3723 00000100:00080000:3.0:1787827273.399545:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827273.399549:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827273.399558:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827273.399568:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972758016:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827273.399576:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972758144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827273.399590:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827273.399594:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827273.399597:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827273.399602:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3723 00000100:00080000:3.0:1787827273.399607:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827273.399612:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827273.399618:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827273.399633:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972758272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827273.399648:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827273.399652:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827273.399654:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827273.399660:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3723 00000100:00100000:1.0:1787827273.399686:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972758144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827273.399725:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972758272:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827273.399744:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827273.399752:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827273.399759:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827273.399904:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972758400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827273.399924:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827273.399965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972758400:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827273.399972:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827273.399976:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827273.399982:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3723 00000100:00080000:3.0:1787827273.399988:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827273.399994:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827273.400001:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827273.400014:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972758528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827273.400031:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827273.400035:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827273.400038:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827273.400043:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3723 00000100:00100000:1.0:1787827273.400048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972758528:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827273.400049:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827273.400057:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827273.400064:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827273.400079:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972758656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827273.400095:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827273.400099:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827273.400102:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827273.400108:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972758656:192.168.203.148@tcp:8:kworker.0 00000100:00080000:3.0:1787827273.400109:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3723 00000100:00080000:3.0:1787827273.400118:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827273.400125:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827273.400131:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827273.400146:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972758784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827273.400175:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972758784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827273.911422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1787827274.039631:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.148:988 00000400:00000100:0.0:1787827274.039672:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827274.039684:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827274.039712:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827274.039718:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827274.039723:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827274.039728:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827274.039733:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827274.039741:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787827274.039768:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827273/real 1787827274] req@ffff9c1a7895fc00 x1874668972758784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827274.039794:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972758784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827274.039822:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827274.039827:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827274.039831:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972758784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827274.039839:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827273/real 1787827274] req@ffff9c1a7895f0c0 x1874668972758656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827274.039852:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972758656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827274.039860:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827274.039864:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827274.039867:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972758656:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827274.039876:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827273/real 1787827274] req@ffff9c1a7895da40 x1874668972758528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827274.039889:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972758528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827274.039897:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827274.039900:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827274.039903:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972758528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827274.039909:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827273/real 1787827274] req@ffff9c1a7895e1c0 x1874668972758400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827274.039920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972758400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827274.039928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827274.039931:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827274.039934:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972758400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827274.039940:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827273/real 1787827274] req@ffff9c1a7895de00 x1874668972758144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827274.039951:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972758144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827274.039959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827274.039963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827274.039966:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972758144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827274.039972:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827273/real 1787827274] req@ffff9c1a7895c780 x1874668972758272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827274.039985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972758272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827274.039993:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827274.039996:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827274.039999:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972758272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827274.040006:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827273/real 1787827274] req@ffff9c1a7895cb40 x1874668972758016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827274.040020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972758016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827274.040028:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827274.040032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827274.040036:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972758016:192.168.203.148@tcp:250:kworker.0 00010000:00010000:1.0:1787827274.167545:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827274.935465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827275.191549:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827275.959445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827276.215481:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827276.983470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827277.239541:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827278.007428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827278.263459:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827278.519426:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827278.519435:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827278.519439:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827278.519453:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3728 00000100:00080000:3.0:1787827278.519462:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827278.519468:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827278.519500:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972758912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827278.519528:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827278.519532:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827278.519534:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827278.519539:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3728 00000100:00080000:3.0:1787827278.519543:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827278.519548:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827278.519558:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827278.519574:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972759040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827278.519591:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972758912:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827278.519608:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827278.519611:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827278.519612:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827278.519616:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3728 00000100:00080000:3.0:1787827278.519619:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827278.519622:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827278.519626:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827278.519638:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972759168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827278.519650:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827278.519656:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827278.519656:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827278.519670:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827278.519677:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3728 00000100:00080000:3.0:1787827278.519686:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827278.519691:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827278.519696:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827278.519701:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827278/real 1787827278] req@ffff9c1a7895d2c0 x1874668972758912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827278.519711:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972759296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827278.519725:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827278.519727:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827278.519729:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827278.519734:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972758912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827278.519734:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3728 00000100:00080000:3.0:1787827278.519740:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827278.519745:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827278.519752:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827278.519763:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972759424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827278.519800:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827278.519803:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827278.519807:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827278.519808:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827278.519813:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827278.519816:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972758912:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827278.519821:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3728 00000100:00080000:3.0:1787827278.519825:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827278.519829:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827278.519833:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827278.519842:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972759552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827278.519845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972759040:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827278.519863:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827278.519866:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827278.519868:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827278.519871:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3728 00000100:00080000:3.0:1787827278.519874:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827278.519876:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827278.519879:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827278.519890:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972759680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827278.519891:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827278.519914:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827278/real 1787827278] req@ffff9c1a7895cb40 x1874668972759040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827278.519927:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972759040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827278.519934:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827278.519936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827278.519938:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972759040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827278.519943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972759168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827278.519959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827278.519966:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827278/real 1787827278] req@ffff9c1a7895c780 x1874668972759168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827278.519974:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972759168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827278.519979:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827278.519981:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827278.519983:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972759168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827278.519987:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972759296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827278.519998:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827278.520004:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827278/real 1787827278] req@ffff9c1a7895de00 x1874668972759296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827278.520011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972759296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827278.520017:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827278.520018:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827278.520020:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972759296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827278.520024:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972759424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827278.520033:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827278.520039:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827278/real 1787827278] req@ffff9c1a7895e1c0 x1874668972759424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827278.520046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972759424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827278.520051:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827278.520052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827278.520054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972759424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827278.520066:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972759552:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827278.520077:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827278.520083:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827278/real 1787827278] req@ffff9c1a7895da40 x1874668972759552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827278.520091:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972759552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827278.520096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827278.520097:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827278.520099:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972759552:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827278.520103:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972759680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827278.520112:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827278.520117:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827278/real 1787827278] req@ffff9c1a7895f0c0 x1874668972759680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827278.520129:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972759680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827278.520137:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827278.520140:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827278.520144:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972759680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827279.031496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827279.287522:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827280.055448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827280.311465:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827281.079486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827281.335542:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827282.103444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827282.359484:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827283.127474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827283.383560:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827283.639480:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827283.639494:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827283.639500:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827283.639513:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3733 00000100:00080000:3.0:1787827283.639525:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827283.639532:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827283.639588:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972759808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827283.639625:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827283.639630:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827283.639632:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827283.639637:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3733 00000100:00080000:3.0:1787827283.639642:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827283.639647:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827283.639658:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827283.639673:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972759936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827283.639682:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827283.639685:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827283.639687:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827283.639691:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3733 00000100:00080000:3.0:1787827283.639693:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827283.639697:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827283.639703:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827283.639705:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972759808:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827283.639715:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972760064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827283.639726:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827283.639730:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827283.639733:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827283.639736:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3733 00000100:00080000:3.0:1787827283.639753:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827283.639758:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827283.639763:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827283.639776:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972760192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827283.639784:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827283.639786:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827283.639788:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827283.639797:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827283.639797:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3733 00000100:00080000:3.0:1787827283.639807:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827283.639811:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827283.639815:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827283.639825:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972760320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827283.639833:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827283.639835:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827283.639837:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827283.639841:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3733 00000100:00080000:3.0:1787827283.639846:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827283.639849:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827283.639855:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827283.639862:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827283/real 1787827283] req@ffff9c1a7895fc00 x1874668972759808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827283.639865:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972760448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827283.639921:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827283.639922:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972759808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827283.639942:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827283.639950:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827283.639962:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3733 00000100:00080000:1.0:1787827283.639964:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827283.639968:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827283.639972:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827283.639979:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827283.639980:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972759808:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787827283.639993:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827283.640001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972759936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827283.640009:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972760576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827283.640045:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827283.640063:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827283/real 1787827283] req@ffff9c1a7895e580 x1874668972759936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827283.640072:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972759936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827283.640078:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827283.640081:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827283.640083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972759936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827283.640103:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972760064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827283.640117:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827283.640124:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827283/real 1787827283] req@ffff9c1a7895f840 x1874668972760064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827283.640132:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972760064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827283.640137:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827283.640139:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827283.640141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972760064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827283.640145:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972760192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827283.640156:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827283.640162:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827283/real 1787827283] req@ffff9c1a7895c000 x1874668972760192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827283.640170:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972760192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827283.640175:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827283.640177:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827283.640179:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972760192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827283.640183:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972760320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827283.640192:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827283.640198:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827283/real 1787827283] req@ffff9c1a7895f480 x1874668972760320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827283.640205:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972760320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827283.640210:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827283.640211:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827283.640213:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972760320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827283.640217:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972760448:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827283.640227:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827283.640233:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827283/real 1787827283] req@ffff9c1a7895cf00 x1874668972760448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827283.640240:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972760448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827283.640245:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827283.640247:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827283.640248:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972760448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827283.640252:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972760576:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827283.640265:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827283.640275:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827283/real 1787827283] req@ffff9c1a7895c3c0 x1874668972760576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827283.640288:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972760576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827283.640296:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827283.640300:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827283.640303:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972760576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827284.151442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827284.407487:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827285.175438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827285.431515:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827286.199498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827286.455587:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827287.223437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827287.479552:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827288.247414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827288.503536:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827288.759614:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827288.759630:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827288.759636:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827288.759651:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3739 00000100:00080000:0.0:1787827288.759666:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827288.759675:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827288.759755:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788603c0 x1874668972760704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827288.759791:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827288.759798:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827288.759812:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827288.759822:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3739 00000100:00080000:0.0:1787827288.759831:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827288.759839:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972760704:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827288.759848:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827288.759868:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827288.759899:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972760832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827288.759922:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827288.759923:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827288.759934:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827288.759938:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827288.759949:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3739 00000100:00080000:0.0:1787827288.759957:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827288.759965:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827288.759977:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827288/real 1787827288] req@ffff9c1a788603c0 x1874668972760704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827288.759988:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827288.760002:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788603c0 x1874668972760704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827288.760018:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972760960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827288.760019:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827288.760032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827288.760044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972760704:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827288.760061:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827288.760068:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827288.760072:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827288.760079:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972760832:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827288.760083:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3739 00000100:00080000:0.0:1787827288.760096:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827288.760106:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827288.760117:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827288.760121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827288.760148:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972761088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827288.760155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827288/real 1787827288] req@ffff9c1a788621c0 x1874668972760832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827288.760210:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827288.760221:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827288.760224:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972760832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827288.760228:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827288.760250:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3739 00000100:00080000:1.0:1787827288.760253:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827288.760278:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827288.760281:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827288.760293:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827288.760296:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972760832:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787827288.760309:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827288.760317:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972760960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827288.760336:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972761216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827288.760355:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827288.760361:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787827288.760365:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827288.760368:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827288.760432:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3739 00000100:00100000:1.0:1787827288.760438:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827288/real 1787827288] req@ffff9c1a78862580 x1874668972760960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827288.760444:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827288.760461:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827288.760464:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972760960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827288.760476:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827288.760479:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827288.760488:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827288.760493:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972760960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827288.760508:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972761344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827288.760517:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972761088:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827288.760544:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827288.760552:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787827288.760557:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827288.760560:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827288.760573:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3739 00000100:00100000:1.0:1787827288.760577:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827288/real 1787827288] req@ffff9c1a78861680 x1874668972761088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827288.760602:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827288.760622:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827288.760626:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972761088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827288.760642:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827288.760645:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827288.760658:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827288.760662:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972761088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827288.760672:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972761216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827288.760683:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972761472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827288.760744:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827288.760783:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827288/real 1787827288] req@ffff9c1a788630c0 x1874668972761216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827288.760797:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972761216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827288.760807:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827288.760811:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827288.760814:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972761216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827288.760828:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972761344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827288.760848:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827288.760858:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827288/real 1787827288] req@ffff9c1a78863c00 x1874668972761344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827288.760870:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972761344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827288.760878:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827288.760881:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827288.760884:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972761344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827288.760889:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972761472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827288.760903:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827288.760912:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827288/real 1787827288] req@ffff9c1a78860000 x1874668972761472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827288.760923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972761472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827288.760930:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827288.760933:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827288.760936:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972761472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827289.271428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827289.527528:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827290.295443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827290.551536:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827291.319414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827291.575568:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827292.343510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827292.599567:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827293.367417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827293.623521:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827293.879451:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827293.879467:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827293.879472:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827293.879480:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3744 00000100:00080000:0.0:1787827293.879489:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827293.879495:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827293.879530:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972761600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827293.879552:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827293.879554:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827293.879556:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827293.879558:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3744 00000100:00080000:0.0:1787827293.879561:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827293.879563:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827293.879570:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827293.879579:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972761728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827293.879587:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827293.879589:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827293.879590:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827293.879593:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3744 00000100:00080000:0.0:1787827293.879595:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827293.879599:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972761600:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827293.879600:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827293.879607:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827293.879617:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972761856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827293.879630:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827293.879633:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827293.879634:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827293.879637:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3744 00000100:00080000:0.0:1787827293.879647:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827293.879649:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827293.879652:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827293.879659:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827293.879663:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972761984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827293.879683:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827293.879686:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827293.879688:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827293.879691:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3744 00000100:00080000:0.0:1787827293.879695:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827293.879698:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827293.879701:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827293.879712:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972762112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827293.879716:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827293/real 1787827293] req@ffff9c1a78860780 x1874668972761600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827293.879725:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827293.879732:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827293.879734:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827293.879738:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3744 00000100:00080000:0.0:1787827293.879743:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827293.879743:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972761600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827293.879751:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827293.879758:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827293.879762:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827293.879767:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972762240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827293.879768:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827293.879776:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972761600:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827293.879783:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827293.879786:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827293.879789:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827293.879789:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972761728:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827293.879797:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3744 00000100:00080000:0.0:1787827293.879799:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827293.879802:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827293.879805:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827293.879814:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972762368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827293.879820:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827293.879837:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827293/real 1787827293] req@ffff9c1a78861e00 x1874668972761728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.879846:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972761728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.879852:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.879855:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827293.879858:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972761728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827293.879876:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972761856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827293.879888:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827293.879894:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827293/real 1787827293] req@ffff9c1a78860000 x1874668972761856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.879901:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972761856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.879907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.879909:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827293.879911:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972761856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827293.879915:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972761984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827293.879925:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827293.879931:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827293/real 1787827293] req@ffff9c1a78863c00 x1874668972761984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.879938:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972761984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.879943:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.879945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827293.879947:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972761984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827293.879951:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972762112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827293.879960:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827293.879965:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827293/real 1787827293] req@ffff9c1a788630c0 x1874668972762112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.879972:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972762112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.879977:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.879979:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827293.879981:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972762112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827293.879984:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972762240:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827293.879994:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827293.880000:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827293/real 1787827293] req@ffff9c1a78861680 x1874668972762240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.880007:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972762240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.880012:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.880014:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827293.880016:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972762240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827293.880019:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972762368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827293.880029:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827293.880034:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827293/real 1787827293] req@ffff9c1a78862580 x1874668972762368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.880043:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972762368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.880052:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.880055:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827293.880059:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972762368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827294.391465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827294.647513:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827295.415406:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827295.671565:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827296.439437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827296.695486:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827297.463461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827297.719578:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827298.487449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827298.743564:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827298.999471:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827298.999479:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827298.999483:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827298.999492:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3749 00000100:00080000:3.0:1787827298.999501:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827298.999507:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827298.999538:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972762496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827298.999562:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827298.999564:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827298.999566:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827298.999569:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3749 00000100:00080000:3.0:1787827298.999571:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827298.999573:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827298.999581:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827298.999592:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972762624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827298.999600:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827298.999603:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827298.999604:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827298.999610:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3749 00000100:00100000:1.0:1787827298.999612:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972762496:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827298.999613:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827298.999619:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827298.999624:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827298.999645:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972762752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827298.999658:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827298.999677:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827298.999679:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827298.999682:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3749 00000100:00080000:3.0:1787827298.999686:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827298.999688:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827298.999692:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827298.999702:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972762880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827298.999707:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827298.999716:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827298.999719:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827298.999720:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827298.999724:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3749 00000100:00080000:3.0:1787827298.999729:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827298.999731:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827298.999735:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827298.999744:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972763008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827298.999750:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827298/real 1787827298] req@ffff9c1a7895e940 x1874668972762496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827298.999754:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827298.999762:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827298.999765:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827298.999772:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3749 00000100:00080000:3.0:1787827298.999776:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827298.999779:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827298.999782:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972762496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827298.999785:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827298.999797:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972763136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827298.999798:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827298.999808:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827298.999811:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827298.999816:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972762496:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827298.999816:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827298.999818:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827298.999825:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3749 00000100:00080000:3.0:1787827298.999829:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827298.999832:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827298.999835:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827298.999842:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972763264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827298.999856:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972762624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827298.999900:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827298.999912:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827298/real 1787827298] req@ffff9c1a7895c3c0 x1874668972762624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827298.999921:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972762624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827298.999930:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827298.999934:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827298.999938:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972762624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827298.999944:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972762752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827298.999965:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827298.999979:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827298/real 1787827298] req@ffff9c1a7895cf00 x1874668972762752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827298.999991:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972762752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827298.999997:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827298.999999:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827299.000001:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972762752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827299.000007:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972762880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827299.000022:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827299.000029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827298/real 1787827299] req@ffff9c1a7895f480 x1874668972762880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827299.000037:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972762880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827299.000043:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827299.000045:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827299.000047:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972762880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827299.000050:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972763008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827299.000060:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827299.000066:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827298/real 1787827299] req@ffff9c1a7895c000 x1874668972763008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827299.000073:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972763008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827299.000078:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827299.000080:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827299.000082:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972763008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827299.000085:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972763136:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827299.000095:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827299.000102:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827298/real 1787827299] req@ffff9c1a7895f840 x1874668972763136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827299.000110:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972763136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827299.000115:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827299.000116:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827299.000118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972763136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827299.000131:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972763264:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827299.000141:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827299.000147:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827298/real 1787827299] req@ffff9c1a7895e580 x1874668972763264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827299.000155:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972763264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827299.000160:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827299.000162:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827299.000164:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972763264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827299.511443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827299.767510:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827300.535410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827300.791567:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827301.559448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827301.815474:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827302.583455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827302.839567:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827303.607468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827303.863569:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827304.119412:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827304.119420:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827304.119425:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827304.119435:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3754 00000100:00080000:3.0:1787827304.119455:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827304.119463:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827304.119498:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972763392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827304.119525:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827304.119528:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827304.119529:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827304.119532:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3754 00000100:00080000:3.0:1787827304.119535:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827304.119538:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827304.119546:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827304.119555:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972763520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827304.119564:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827304.119569:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827304.119570:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972763392:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827304.119571:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827304.119579:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3754 00000100:00080000:3.0:1787827304.119582:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827304.119584:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827304.119587:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827304.119606:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972763648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827304.119615:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827304.119618:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827304.119620:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827304.119623:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3754 00000100:00080000:3.0:1787827304.119628:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827304.119631:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827304.119634:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827304.119645:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972763776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827304.119651:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827304.119657:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827304.119660:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827304.119662:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827304.119666:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3754 00000100:00080000:3.0:1787827304.119674:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827304.119677:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827304.119681:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827304.119691:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827304/real 1787827304] req@ffff9c1a7895fc00 x1874668972763392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827304.119695:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972763904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827304.119729:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827304.119731:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827304.119732:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827304.119735:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3754 00000100:00080000:3.0:1787827304.119737:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827304.119740:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972763392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827304.119743:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827304.119751:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827304.119757:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827304.119761:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972764032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827304.119765:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827304.119774:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972763392:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827304.119776:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827304.119781:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827304.119782:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827304.119786:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3754 00000100:00080000:3.0:1787827304.119789:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827304.119792:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827304.119795:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827304.119800:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972763520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827304.119804:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972764160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827304.119829:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827304.119841:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827304/real 1787827304] req@ffff9c1a7895f0c0 x1874668972763520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827304.119850:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972763520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827304.119856:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827304.119858:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827304.119861:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972763520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827304.119865:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972763648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827304.119877:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827304.119883:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827304/real 1787827304] req@ffff9c1a7895da40 x1874668972763648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827304.119890:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972763648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827304.119896:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827304.119898:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827304.119900:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972763648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827304.119904:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972763776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827304.119922:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827304.119933:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827304/real 1787827304] req@ffff9c1a7895e1c0 x1874668972763776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827304.119946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972763776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827304.119956:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827304.119959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827304.119962:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972763776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827304.119967:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972763904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827304.119982:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827304.119998:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827304/real 1787827304] req@ffff9c1a7895de00 x1874668972763904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827304.120006:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972763904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827304.120013:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827304.120016:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827304.120019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972763904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827304.120026:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972764032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827304.120048:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827304.120061:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827304/real 1787827304] req@ffff9c1a7895c780 x1874668972764032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827304.120075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972764032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827304.120085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827304.120089:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827304.120092:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972764032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827304.120112:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972764160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827304.120134:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827304.120145:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827304/real 1787827304] req@ffff9c1a7895cb40 x1874668972764160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827304.120159:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972764160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827304.120168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827304.120172:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827304.120175:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972764160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827304.631455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827304.887514:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827305.655451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827305.911532:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827306.679473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827306.935525:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827307.703500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827307.959541:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827308.727442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827308.983535:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827309.239655:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972764288:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827309.239857:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972764416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827309.240021:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972764544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827309.240112:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972764672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827309.240187:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972764800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827309.240203:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972764928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827309.240217:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972765056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827309.751448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827310.007572:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827310.775424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1787827310.903638:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787827310.903682:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827310.903698:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827310.903734:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827310.903758:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827310.903772:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787827310.903772:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827309/real 1787827310] req@ffff9c1a49d8f0c0 x1874668972764416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787827310.903784:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827310.903795:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827310.903807:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787827310.903824:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972764416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827310.903838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827310.903843:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827310.903849:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972764416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827310.903857:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827309/real 1787827310] req@ffff9c1a49d8d680 x1874668972764544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827310.903872:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972764544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827310.903880:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827310.903884:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827310.903888:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972764544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827310.903895:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827309/real 1787827310] req@ffff9c1a49d8f480 x1874668972764672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827310.903913:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f480 x1874668972764672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827310.903921:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827310.903924:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827310.903927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972764672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827310.903934:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827309/real 1787827310] req@ffff9c1a49d8c000 x1874668972764288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827310.903945:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972764288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827310.903953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827310.903956:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827310.903959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972764288:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827310.903984:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827309/real 1787827310] req@ffff9c1a49d8de00 x1874668972764800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827310.903997:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972764800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827310.904004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827310.904007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827310.904010:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972764800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827310.904017:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827309/real 1787827310] req@ffff9c1a49d8da40 x1874668972764928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827310.904029:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972764928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827310.904037:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827310.904048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827310.904051:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972764928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827310.904058:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827309/real 1787827310] req@ffff9c1a49d8cf00 x1874668972765056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827310.904076:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972765056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827310.904084:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827310.904087:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827310.904091:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972765056:192.168.203.148@tcp:8:kworker.0 00010000:00010000:0.0:1787827311.031494:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827311.799469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827312.055538:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827312.823402:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827313.079565:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827313.847437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827314.103515:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827314.359623:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972765184:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787827314.359692:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827314.359747:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827314/real 1787827314] req@ffff9c1a49d8c780 x1874668972765184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827314.359775:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972765184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827314.359830:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827314.359838:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827314.359845:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972765184:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827314.359875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972765312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827314.359919:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827314.359937:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827314/real 1787827314] req@ffff9c1a49d8cf00 x1874668972765312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827314.359947:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972765312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827314.359954:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827314.359957:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827314.359959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972765312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827314.359981:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972765440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827314.359995:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827314.360002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827314/real 1787827314] req@ffff9c1a49d8da40 x1874668972765440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827314.360010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972765440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827314.360015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827314.360017:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827314.360019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972765440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827314.360023:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972765568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827314.360034:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827314.360044:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827314/real 1787827314] req@ffff9c1a49d8de00 x1874668972765568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827314.360056:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972765568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827314.360065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827314.360068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827314.360072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972765568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827314.360079:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972765696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827314.360097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827314.360105:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827314/real 1787827314] req@ffff9c1a49d8c000 x1874668972765696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827314.360113:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972765696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827314.360118:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827314.360120:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827314.360122:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972765696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827314.360153:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972765824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827314.360167:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827314.360175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827314/real 1787827314] req@ffff9c1a49d8f480 x1874668972765824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827314.360183:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f480 x1874668972765824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827314.360188:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827314.360190:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827314.360193:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972765824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827314.360196:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972765952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827314.360206:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827314.360212:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827314/real 1787827314] req@ffff9c1a49d8d680 x1874668972765952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827314.360220:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972765952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827314.360225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827314.360227:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827314.360229:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972765952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827314.871410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827315.127496:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827315.895440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827316.151561:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827316.919456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827317.175555:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827317.943459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827318.199602:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827318.967454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827319.223481:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827319.479574:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827319.479587:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827319.479592:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827319.479602:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3769 00000100:00080000:3.0:1787827319.479612:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827319.479618:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827319.479659:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972766080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827319.479699:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827319.479703:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827319.479705:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827319.479727:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3769 00000100:00080000:3.0:1787827319.479732:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827319.479736:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827319.479746:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827319.479763:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972766208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827319.479778:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972766080:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827319.479778:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827319.479789:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827319.479796:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827319.479801:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3769 00000100:00080000:3.0:1787827319.479805:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827319.479808:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827319.479813:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827319.479822:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972766336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827319.479846:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827319.479850:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827319.479851:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827319.479855:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3769 00000100:00080000:3.0:1787827319.479857:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827319.479860:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827319.479863:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827319.479870:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972766464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827319.479877:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827319.479880:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827319.479882:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827319.479887:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3769 00000100:00080000:3.0:1787827319.479890:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827319.479895:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827319.479900:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827319.479928:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972766592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827319.479940:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827319.479944:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827319.479947:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827319.479952:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3769 00000100:00080000:3.0:1787827319.479960:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827319.479965:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827319.479972:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827319.479985:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972766720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827319.479993:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827319.480000:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827319.480004:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827319.480006:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827319.480011:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3769 00000100:00080000:3.0:1787827319.480024:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827319.480028:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827319.480033:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827319.480043:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827319/real 1787827319] req@ffff9c1a7895d2c0 x1874668972766080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827319.480046:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972766848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827319.480084:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972766080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.480096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.480100:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827319.480105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972766080:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827319.480129:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972766208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827319.480154:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827319.480164:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827319/real 1787827319] req@ffff9c1a7895cb40 x1874668972766208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.480175:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972766208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.480182:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.480185:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827319.480189:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972766208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827319.480193:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972766336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827319.480207:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827319.480214:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827319/real 1787827319] req@ffff9c1a7895c780 x1874668972766336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.480223:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972766336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.480230:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.480232:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827319.480234:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972766336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827319.480239:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972766464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827319.480251:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827319.480258:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827319/real 1787827319] req@ffff9c1a7895de00 x1874668972766464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.480267:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972766464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.480273:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.480276:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827319.480278:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972766464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827319.480282:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972766592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827319.480294:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827319.480301:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827319/real 1787827319] req@ffff9c1a7895e1c0 x1874668972766592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.480310:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972766592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.480316:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.480318:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827319.480321:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972766592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827319.480325:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972766720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827319.480338:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827319.480345:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827319/real 1787827319] req@ffff9c1a7895da40 x1874668972766720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.480353:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972766720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.480359:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.480412:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827319.480417:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972766720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827319.480425:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972766848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827319.480451:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827319.480465:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827319/real 1787827319] req@ffff9c1a7895f0c0 x1874668972766848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827319.480479:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972766848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827319.480532:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827319.480535:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827319.480538:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972766848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827319.991444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827320.247493:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827321.015461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827321.271544:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827322.039453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827322.295547:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827323.063428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827323.319522:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827324.087449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827324.343500:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827324.599495:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827324.599506:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827324.599511:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827324.599522:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3774 00000100:00080000:3.0:1787827324.599536:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827324.599543:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827324.599587:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972766976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827324.599616:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827324.599620:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827324.599622:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827324.599627:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3774 00000100:00080000:3.0:1787827324.599630:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827324.599635:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827324.599648:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827324.599662:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972767104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827324.599672:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827324.599674:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972766976:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827324.599675:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827324.599677:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827324.599681:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3774 00000100:00080000:3.0:1787827324.599684:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827324.599688:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827324.599692:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827324.599701:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972767232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827324.599710:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827324.599714:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827324.599715:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827324.599719:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3774 00000100:00080000:3.0:1787827324.599722:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827324.599725:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827324.599730:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827324.599739:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972767360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827324.599747:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827324.599750:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827324.599752:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827324.599756:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827324.599756:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3774 00000100:00080000:3.0:1787827324.599759:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827324.599762:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827324.599766:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827324.599775:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972767488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827324.599782:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827324.599785:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827324.599788:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827324.599791:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3774 00000100:00080000:3.0:1787827324.599794:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827324.599798:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827324.599803:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827324.599806:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827324/real 1787827324] req@ffff9c1a7895fc00 x1874668972766976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827324.599812:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972767616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827324.599821:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827324.599824:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827324.599826:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827324.599827:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972766976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827324.599830:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3774 00000100:00080000:3.0:1787827324.599833:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827324.599836:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827324.599839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827324.599841:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827324.599844:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827324.599848:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972766976:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827324.599849:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972767744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827324.599876:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972767104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827324.599901:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827324.599916:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827324/real 1787827324] req@ffff9c1a7895e580 x1874668972767104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827324.599926:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972767104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827324.599932:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827324.599934:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827324.599937:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972767104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827324.599942:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972767232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827324.599960:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827324.599972:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827324/real 1787827324] req@ffff9c1a7895f840 x1874668972767232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827324.599983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972767232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827324.599991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827324.599994:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827324.599997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972767232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827324.600003:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972767360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827324.600023:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827324.600047:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827324/real 1787827324] req@ffff9c1a7895c000 x1874668972767360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827324.600059:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972767360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827324.600067:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827324.600071:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827324.600074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972767360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827324.600085:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972767488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827324.600107:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827324.600119:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827324/real 1787827324] req@ffff9c1a7895f480 x1874668972767488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827324.600132:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972767488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827324.600141:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827324.600144:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827324.600146:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972767488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827324.600151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972767616:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827324.600169:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827324.600178:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827324/real 1787827324] req@ffff9c1a7895cf00 x1874668972767616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827324.600186:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972767616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827324.600191:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827324.600194:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827324.600196:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972767616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827324.600199:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972767744:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827324.600209:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827324.600214:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827324/real 1787827324] req@ffff9c1a7895c3c0 x1874668972767744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827324.600222:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972767744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827324.600227:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827324.600230:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827324.600232:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972767744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827325.111460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827325.367529:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827326.135466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827326.391523:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827327.159465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827327.415565:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827328.183424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827328.439562:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827329.207547:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827329.463557:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827329.719497:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827329.719522:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827329.719526:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827329.719535:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3780 00000100:00080000:1.0:1787827329.719545:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827329.719551:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827329.719629:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972767872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827329.719661:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827329.719666:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827329.719668:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827329.719673:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3780 00000100:00080000:1.0:1787827329.719678:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827329.719683:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827329.719693:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827329.719705:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972767872:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827329.719712:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972768000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827329.719726:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827329.719729:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827329.719731:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827329.719733:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3780 00000100:00080000:1.0:1787827329.719736:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827329.719739:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827329.719743:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827329.719753:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972768128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827329.719762:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827329.719764:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827329.719765:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827329.719767:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3780 00000100:00080000:1.0:1787827329.719769:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827329.719771:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827329.719775:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827329.719783:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972768256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827329.719787:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827329.719796:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827329.719799:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827329.719800:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827329.719804:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3780 00000100:00080000:1.0:1787827329.719807:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827329.719810:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827329.719813:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827329.719820:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972768384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827329.719827:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827329.719828:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827329/real 1787827329] req@ffff9c1a78a79e00 x1874668972767872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827329.719832:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827329.719835:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827329.719842:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3780 00000100:00080000:1.0:1787827329.719846:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827329.719866:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827329.719871:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827329.719874:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972767872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827329.719881:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972768512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827329.719911:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827329.719912:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827329.719914:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827329.719916:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827329.719918:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3780 00000100:00080000:1.0:1787827329.719921:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827329.719925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827329.719926:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827329.719931:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827329.719932:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972767872:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827329.719941:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972768640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827329.719957:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972768000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827329.719977:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827329.719986:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827329/real 1787827329] req@ffff9c1a78a7b0c0 x1874668972768000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827329.719995:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972768000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827329.720000:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827329.720003:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827329.720005:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972768000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827329.720009:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972768128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827329.720019:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827329.720025:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827329/real 1787827329] req@ffff9c1a78a7bc00 x1874668972768128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827329.720032:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972768128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827329.720037:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827329.720039:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827329.720041:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972768128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827329.720045:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972768256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827329.720055:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827329.720061:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827329/real 1787827329] req@ffff9c1a78a78780 x1874668972768256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827329.720069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972768256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827329.720074:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827329.720076:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827329.720078:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972768256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827329.720081:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972768384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827329.720090:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827329.720096:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827329/real 1787827329] req@ffff9c1a78a7a1c0 x1874668972768384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827329.720103:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972768384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827329.720108:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827329.720109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827329.720111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972768384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827329.720114:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972768512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827329.720124:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827329.720130:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827329/real 1787827329] req@ffff9c1a78a783c0 x1874668972768512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827329.720148:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972768512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827329.720154:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827329.720155:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827329.720157:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972768512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827329.720161:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972768640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827329.720171:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827329.720177:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827329/real 1787827329] req@ffff9c1a78a7a940 x1874668972768640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827329.720188:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972768640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827329.720193:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827329.720195:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827329.720197:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972768640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827330.231482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827330.487529:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827331.255463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827331.511529:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827332.279428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827332.535483:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827333.303464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827333.559537:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827334.327431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827334.583531:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827334.839502:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827334.839513:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827334.839517:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827334.839527:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3785 00000100:00080000:3.0:1787827334.839538:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827334.839545:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827334.839580:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972768768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827334.839607:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827334.839611:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827334.839613:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827334.839618:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3785 00000100:00080000:3.0:1787827334.839621:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827334.839626:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827334.839637:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827334.839652:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972768768:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827334.839653:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972768896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827334.839682:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827334.839685:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827334.839688:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827334.839692:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3785 00000100:00080000:3.0:1787827334.839698:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827334.839702:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827334.839707:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827334.839720:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972769024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827334.839724:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827334.839743:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827334.839746:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827334.839748:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827334.839753:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3785 00000100:00080000:3.0:1787827334.839757:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827334.839761:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827334.839767:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827334.839769:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827334/real 1787827334] req@ffff9c1a7895e940 x1874668972768768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827334.839780:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972769152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827334.839790:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827334.839793:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972768768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827334.839795:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827334.839798:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827334.839807:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3785 00000100:00080000:3.0:1787827334.839813:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827334.839814:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827334.839818:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827334.839824:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787827334.839825:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827334.839831:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972768768:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827334.839838:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972769280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827334.839849:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827334.839852:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827334.839855:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827334.839859:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972768896:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827334.839863:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3785 00000100:00080000:3.0:1787827334.839870:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827334.839875:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827334.839881:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827334.839894:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972769408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827334.839901:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827334.839912:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827334.839916:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827334.839918:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827334.839923:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827334/real 1787827334] req@ffff9c1a7895c3c0 x1874668972768896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827334.839925:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3785 00000100:00080000:3.0:1787827334.839933:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827334.839941:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827334.839947:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827334.839950:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972768896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827334.839960:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827334.839965:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827334.839969:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972769536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827334.839970:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972768896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827334.839984:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972769024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827334.840009:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827334.840021:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827334/real 1787827334] req@ffff9c1a7895cf00 x1874668972769024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827334.840038:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972769024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827334.840046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827334.840049:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827334.840053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972769024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827334.840059:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972769152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827334.840078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827334.840089:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827334/real 1787827334] req@ffff9c1a7895f480 x1874668972769152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827334.840101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972769152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827334.840107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827334.840109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827334.840111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972769152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827334.840115:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972769280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827334.840129:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827334.840135:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827334/real 1787827334] req@ffff9c1a7895c000 x1874668972769280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827334.840143:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972769280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827334.840152:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827334.840154:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827334.840156:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972769280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827334.840169:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972769408:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827334.840180:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827334.840185:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827334/real 1787827334] req@ffff9c1a7895f840 x1874668972769408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827334.840193:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972769408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827334.840198:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827334.840199:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827334.840201:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972769408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827334.840205:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972769536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827334.840214:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827334.840219:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827334/real 1787827334] req@ffff9c1a7895e580 x1874668972769536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827334.840234:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972769536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827334.840239:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827334.840241:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827334.840244:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972769536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827335.351448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827335.607538:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827336.375439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827336.631519:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827337.399467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827337.655522:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827338.423464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827338.679511:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827339.447454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827339.703518:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827339.959450:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827339.959462:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827339.959466:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827339.959476:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3790 00000100:00080000:0.0:1787827339.959487:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827339.959494:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827339.959530:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972769664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827339.959555:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827339.959558:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827339.959559:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827339.959562:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3790 00000100:00080000:0.0:1787827339.959565:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827339.959568:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827339.959576:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827339.959588:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788603c0 x1874668972769792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827339.959597:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972769664:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827339.959598:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827339.959606:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827339.959608:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827339.959611:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3790 00000100:00080000:0.0:1787827339.959613:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827339.959616:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827339.959619:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827339.959628:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972769920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827339.959636:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827339.959639:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827339.959640:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827339.959642:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3790 00000100:00080000:0.0:1787827339.959645:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827339.959648:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827339.959651:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827339.959658:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827339.959662:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972770048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827339.959671:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827339.959673:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827339.959675:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827339.959677:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3790 00000100:00080000:0.0:1787827339.959679:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827339.959681:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827339.959685:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827339.959693:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827339/real 1787827339] req@ffff9c1a788621c0 x1874668972769664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827339.959699:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972770176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827339.959713:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972769664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827339.959714:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827339.959725:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827339.959730:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827339.959733:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827339.959738:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3790 00000100:00080000:1.0:1787827339.959739:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827339.959745:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827339.959748:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972769664:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827339.959752:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827339.959760:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827339.959770:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972770304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827339.959778:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827339.959783:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787827339.959783:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972769792:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827339.959787:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827339.959795:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3790 00000100:00080000:0.0:1787827339.959799:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827339.959803:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827339.959809:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827339.959819:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827339.959825:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972770432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827339.959834:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827339/real 1787827339] req@ffff9c1a788603c0 x1874668972769792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827339.959849:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788603c0 x1874668972769792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827339.959855:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827339.959858:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827339.959863:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972769792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827339.959868:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972769920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827339.959883:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827339.959890:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827339/real 1787827339] req@ffff9c1a78862940 x1874668972769920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827339.959898:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972769920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827339.959903:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827339.959905:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827339.959907:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972769920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827339.959910:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972770048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827339.959920:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827339.959926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827339/real 1787827339] req@ffff9c1a78860b40 x1874668972770048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827339.959933:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972770048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827339.959938:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827339.959939:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827339.959941:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972770048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827339.959945:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972770176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827339.959954:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827339.959959:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827339/real 1787827339] req@ffff9c1a78861a40 x1874668972770176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827339.959971:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972770176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827339.959976:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827339.959978:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827339.959979:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972770176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827339.959991:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972770304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827339.960002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827339.960008:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827339/real 1787827339] req@ffff9c1a78860f00 x1874668972770304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827339.960016:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972770304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827339.960030:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827339.960032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827339.960034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972770304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827339.960038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972770432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827339.960049:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827339.960055:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827339/real 1787827339] req@ffff9c1a78862d00 x1874668972770432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827339.960062:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972770432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827339.960072:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827339.960074:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827339.960076:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972770432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827340.471446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827340.727484:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827341.495433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827341.751518:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827342.519450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827342.775603:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827343.543513:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827343.799497:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827344.567435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827344.823510:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827345.079455:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827345.079464:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827345.079469:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827345.079479:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3795 00000100:00080000:0.0:1787827345.079491:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827345.079498:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827345.079529:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972770560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827345.079549:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827345.079552:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827345.079553:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827345.079557:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3795 00000100:00080000:0.0:1787827345.079560:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827345.079562:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827345.079571:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827345.079578:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972770560:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827345.079585:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972770688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827345.079594:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827345.079597:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827345.079599:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827345.079603:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3795 00000100:00080000:0.0:1787827345.079606:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827345.079611:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827345.079617:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827345.079632:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972770816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827345.079642:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827345.079658:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827345.079663:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827345.079665:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827345.079670:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3795 00000100:00080000:0.0:1787827345.079675:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827345.079679:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827345.079683:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827345/real 1787827345] req@ffff9c1a788612c0 x1874668972770560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827345.079684:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827345.079703:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972770944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827345.079706:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972770560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827345.079726:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827345.079729:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827345.079731:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827345.079735:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827345.079735:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827345.079745:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972770560:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827345.079746:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3795 00000100:00080000:0.0:1787827345.079753:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827345.079757:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827345.079761:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827345.079770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972770688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827345.079776:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972771072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827345.079791:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827345.079795:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827345.079800:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827345.079803:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827345.079810:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3795 00000100:00100000:1.0:1787827345.079813:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827345/real 1787827345] req@ffff9c1a78863840 x1874668972770688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827345.079816:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827345.079824:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827345.079831:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827345.079832:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972770688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827345.079841:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827345.079846:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827345.079847:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972771200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827345.079859:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827345.079860:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972770688:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827345.079868:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787827345.079872:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972770816:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827345.079873:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827345.079882:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3795 00000100:00080000:0.0:1787827345.079889:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827345.079893:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827345.079898:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827345.079905:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827345/real 1787827345] req@ffff9c1a78862d00 x1874668972770816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827345.079906:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827345.079919:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972770816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827345.079924:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972771328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827345.079931:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827345.079938:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827345.079943:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972770816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827345.079949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972770944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827345.079963:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827345.079972:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827345/real 1787827345] req@ffff9c1a78860f00 x1874668972770944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827345.079983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972770944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827345.079989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827345.079991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827345.079993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972770944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827345.080007:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972771072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827345.080025:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827345.080042:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827345/real 1787827345] req@ffff9c1a78861a40 x1874668972771072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827345.080057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972771072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827345.080066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827345.080070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827345.080074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972771072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827345.080081:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972771200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827345.080097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827345.080106:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827345/real 1787827345] req@ffff9c1a78860b40 x1874668972771200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827345.080114:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972771200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827345.080119:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827345.080121:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827345.080123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972771200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827345.080126:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972771328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827345.080136:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827345.080141:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827345/real 1787827345] req@ffff9c1a78862940 x1874668972771328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827345.080149:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972771328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827345.080154:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827345.080156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827345.080158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972771328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827345.591432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827345.847489:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827346.615491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827346.871546:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827347.639420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827347.895495:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827348.663443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827348.919644:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827349.687427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827349.943622:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827350.199433:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827350.199442:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827350.199447:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827350.199458:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3800 00000100:00080000:0.0:1787827350.199470:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827350.199479:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827350.199514:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788603c0 x1874668972771456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827350.199539:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827350.199541:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827350.199543:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827350.199546:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3800 00000100:00080000:0.0:1787827350.199549:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827350.199552:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827350.199560:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827350.199574:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972771584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827350.199574:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972771456:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827350.199588:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827350.199591:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827350.199592:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827350.199595:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3800 00000100:00080000:0.0:1787827350.199597:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827350.199599:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827350.199602:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827350.199611:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972771712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827350.199618:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827350.199621:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827350.199622:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827350.199625:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3800 00000100:00080000:0.0:1787827350.199628:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827350.199631:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827350.199634:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827350.199638:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827350.199643:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972771840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827350.199651:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827350.199654:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827350.199655:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827350.199658:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3800 00000100:00080000:0.0:1787827350.199660:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827350.199662:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827350.199665:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827350.199675:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972771968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827350.199679:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827350/real 1787827350] req@ffff9c1a788603c0 x1874668972771456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827350.199688:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827350.199696:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827350.199698:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827350.199702:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3800 00000100:00080000:0.0:1787827350.199704:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827350.199706:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788603c0 x1874668972771456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827350.199710:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827350.199716:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827350.199724:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827350.199728:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972772096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827350.199731:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827350.199739:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972771456:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827350.199753:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827350.199754:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827350.199756:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827350.199759:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3800 00000100:00080000:0.0:1787827350.199761:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827350.199763:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827350.199767:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827350.199768:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972771584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827350.199777:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972772224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827350.199792:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827350.199804:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827350/real 1787827350] req@ffff9c1a788621c0 x1874668972771584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827350.199812:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972771584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827350.199818:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827350.199821:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827350.199823:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972771584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827350.199827:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972771712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827350.199837:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827350.199843:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827350/real 1787827350] req@ffff9c1a78862580 x1874668972771712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827350.199851:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972771712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827350.199856:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827350.199857:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827350.199859:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972771712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827350.199863:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972771840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827350.199872:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827350.199878:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827350/real 1787827350] req@ffff9c1a78861680 x1874668972771840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827350.199885:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972771840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827350.199890:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827350.199891:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827350.199893:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972771840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827350.199896:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972771968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827350.199906:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827350.199912:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827350/real 1787827350] req@ffff9c1a788630c0 x1874668972771968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827350.199919:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972771968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827350.199923:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827350.199925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827350.199927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972771968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827350.199930:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972772096:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827350.199940:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827350.199946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827350/real 1787827350] req@ffff9c1a78863c00 x1874668972772096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827350.199953:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972772096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827350.199958:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827350.199959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827350.199961:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972772096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827350.199972:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972772224:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827350.199983:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827350.199989:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827350/real 1787827350] req@ffff9c1a78860000 x1874668972772224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827350.200001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972772224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827350.200006:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827350.200007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827350.200009:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972772224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827350.711442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827350.711467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827350.967569:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827351.735462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827351.735462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827351.991520:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827352.759460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827352.759460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827353.015512:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827353.783444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827353.783481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827354.039533:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827354.807449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827354.807449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827355.063516:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827355.319671:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972772352:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787827355.319765:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827355.319832:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827355/real 1787827355] req@ffff9c1a49d8f0c0 x1874668972772352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827355.319874:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972772352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827355.319889:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827355.319895:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827355.319900:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972772352:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827355.319912:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972772480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827355.319945:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827355.319964:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827355/real 1787827355] req@ffff9c1a49d8e940 x1874668972772480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827355.319982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e940 x1874668972772480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827355.319994:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827355.319999:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827355.320003:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972772480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827355.320049:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972772608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827355.320094:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827355.320110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827355/real 1787827355] req@ffff9c1a49d8cb40 x1874668972772608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827355.320125:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972772608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827355.320152:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827355.320156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827355.320161:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972772608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827355.320169:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972772736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827355.320194:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827355.320209:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827355/real 1787827355] req@ffff9c1a49d8ed00 x1874668972772736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827355.320223:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972772736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827355.320234:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827355.320258:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827355.320265:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972772736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827355.320273:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972772864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827355.320301:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827355.320317:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827355/real 1787827355] req@ffff9c1a49d8c3c0 x1874668972772864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827355.320333:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972772864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827355.320377:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827355.320383:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827355.320388:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972772864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827355.320398:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972772992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827355.320427:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827355.320443:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827355/real 1787827355] req@ffff9c1a49d8e580 x1874668972772992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827355.320459:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972772992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827355.320469:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827355.320474:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827355.320478:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972772992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827355.320486:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972773120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827355.320511:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827355.320527:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827355/real 1787827355] req@ffff9c1a49d8fc00 x1874668972773120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827355.320544:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972773120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827355.320555:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827355.320559:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827355.320563:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972773120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827355.831421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827355.831421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827356.087507:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827356.855423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827356.855442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827357.111536:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827357.879444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827357.879444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827358.135490:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827358.903459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827359.159551:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827359.927456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827360.183477:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827360.439533:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827360.439547:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827360.439553:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827360.439567:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3810 00000100:00080000:0.0:1787827360.439581:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827360.439593:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827360.439640:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972773248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827360.439682:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827360.439689:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827360.439692:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827360.439700:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3810 00000100:00080000:0.0:1787827360.439707:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827360.439714:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827360.439729:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827360.439742:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972773248:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827360.439755:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972773376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827360.439778:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827360.439801:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827360.439805:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827360.439811:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3810 00000100:00080000:0.0:1787827360.439820:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827360.439826:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787827360.439833:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787827360.439835:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827360.439860:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972773504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827360.439877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827360/real 1787827360] req@ffff9c1a78861e00 x1874668972773248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827360.439878:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827360.439892:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827360.439914:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827360.439925:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3810 00000100:00100000:1.0:1787827360.439930:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972773248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827360.439936:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827360.439950:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827360.439952:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827360.439962:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787827360.439967:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827360.439980:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972773248:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827360.439997:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972773632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827360.440011:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972773376:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827360.440040:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827360.440047:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827360.440051:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827360.440059:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3810 00000800:00000100:1.0:1787827360.440065:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827360.440071:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827360.440083:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827360.440085:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827360/real 1787827360] req@ffff9c1a78860780 x1874668972773376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827360.440116:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827360.440123:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972773376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827360.440135:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827360.440140:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827360.440148:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972773760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827360.440149:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972773376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827360.440166:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972773504:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827360.440188:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827360.440196:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787827360.440200:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827360.440225:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827360.440237:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3810 00000100:00080000:0.0:1787827360.440246:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827360.440247:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827360/real 1787827360] req@ffff9c1a78860000 x1874668972773504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827360.440261:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827360.440270:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972773504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827360.440275:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827360.440281:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827360.440286:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827360.440290:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972773504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827360.440306:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972773888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827360.440312:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972773632:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827360.440340:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827360.440386:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827360.440409:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827360/real 1787827360] req@ffff9c1a78863c00 x1874668972773632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827360.440412:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827360.440420:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827360.440431:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972773632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827360.440434:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3810 00000100:00080000:0.0:1787827360.440448:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827360.440450:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827360.440459:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827360.440481:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827360.440485:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972773632:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787827360.440496:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827360.440497:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972773760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827360.440545:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972774016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827360.440550:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827360.440569:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827360/real 1787827360] req@ffff9c1a788630c0 x1874668972773760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827360.440584:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972773760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827360.440593:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827360.440596:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827360.440600:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972773760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827360.440617:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972773888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827360.440635:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827360.440644:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827360/real 1787827360] req@ffff9c1a78861680 x1874668972773888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827360.440656:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972773888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827360.440665:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827360.440669:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827360.440674:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972773888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827360.440683:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972774016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827360.440714:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827360.440732:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827360/real 1787827360] req@ffff9c1a78862580 x1874668972774016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827360.440751:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972774016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827360.440760:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827360.440763:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827360.440766:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972774016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827360.951443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827361.207522:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827361.975441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827362.231489:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827362.999480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827363.255635:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827364.023487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827364.279591:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827365.047464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827365.303598:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827365.559601:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972774144:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787827365.559710:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827365.559770:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827365/real 1787827365] req@ffff9c1a49d8e1c0 x1874668972774144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827365.559829:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972774144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827365.559879:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827365.559886:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827365.559892:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972774144:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827365.559903:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972774272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827365.559934:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827365.559950:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827365/real 1787827365] req@ffff9c1a49d8fc00 x1874668972774272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827365.559984:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972774272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827365.559994:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827365.559998:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827365.560002:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972774272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827365.560035:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972774400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827365.560059:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827365.560071:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827365/real 1787827365] req@ffff9c1a49d8e580 x1874668972774400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827365.560085:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972774400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827365.560093:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827365.560097:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827365.560101:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972774400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827365.560109:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972774528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827365.560131:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827365.560144:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827365/real 1787827365] req@ffff9c1a49d8c3c0 x1874668972774528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827365.560157:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972774528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827365.560188:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827365.560192:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827365.560195:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972774528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827365.560203:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972774656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827365.560226:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827365.560239:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827365/real 1787827365] req@ffff9c1a49d8ed00 x1874668972774656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827365.560253:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972774656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827365.560263:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827365.560267:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827365.560270:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972774656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827365.560278:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972774784:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827365.560300:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827365.560313:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827365/real 1787827365] req@ffff9c1a49d8cb40 x1874668972774784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827365.560327:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972774784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827365.560336:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827365.560340:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827365.560372:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972774784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827365.560379:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972774912:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827365.560399:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827365.560410:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827365/real 1787827365] req@ffff9c1a49d8e940 x1874668972774912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827365.560423:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e940 x1874668972774912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827365.560432:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827365.560435:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827365.560439:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972774912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827366.071484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827366.327501:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827367.095427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827367.351497:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827368.119408:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827368.375587:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827369.143465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827369.399630:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827370.167475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827370.423480:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827370.679599:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972775040:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827370.679702:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972775168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827370.679749:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972775296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827370.679771:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972775424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827370.679789:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972775552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827370.679806:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972775680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827370.679825:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972775808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827371.191475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827371.447586:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827372.215478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827372.471524:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827373.239417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827373.495529:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787827373.751722:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787827373.751788:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827373.751805:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827373.751846:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827373.751857:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827373.751869:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827373.751882:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787827373.751885:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827370/real 1787827373] req@ffff9c1a49d8f480 x1874668972775296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787827373.751893:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827373.751904:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787827373.751938:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f480 x1874668972775296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827373.751950:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827373.751955:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827373.751959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972775296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827373.751967:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827370/real 1787827373] req@ffff9c1a49d8c000 x1874668972775424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827373.751979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972775424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827373.751986:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827373.751990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827373.751993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972775424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827373.752000:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827370/real 1787827373] req@ffff9c1a49d8de00 x1874668972775552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827373.752011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972775552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827373.752019:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827373.752023:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827373.752026:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972775552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827373.752033:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827370/real 1787827373] req@ffff9c1a49d8da40 x1874668972775680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827373.752046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972775680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827373.752060:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827373.752062:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827373.752064:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972775680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827373.752068:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827370/real 1787827373] req@ffff9c1a49d8cf00 x1874668972775808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827373.752075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972775808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827373.752080:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827373.752082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827373.752084:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972775808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827373.752088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827370/real 1787827373] req@ffff9c1a49d8f0c0 x1874668972775040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827373.752100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972775040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827373.752108:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827373.752134:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827373.752137:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972775040:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827373.752144:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827370/real 1787827373] req@ffff9c1a49d8d680 x1874668972775168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827373.752158:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972775168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827373.752177:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827373.752180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827373.752183:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972775168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827374.263458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827374.519572:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827375.287433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827375.543481:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827375.799615:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972775936:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787827375.799713:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827375.799761:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827375/real 1787827375] req@ffff9c1a49d8c780 x1874668972775936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827375.799811:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972775936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827375.799831:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827375.799840:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827375.799851:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972775936:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827375.799886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972776064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827375.799933:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827375.799961:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827375/real 1787827375] req@ffff9c1a49d8d680 x1874668972776064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827375.799982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972776064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827375.799999:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827375.800009:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827375.800016:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972776064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827375.800027:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972776192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827375.800052:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827375.800063:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827375/real 1787827375] req@ffff9c1a49d8f0c0 x1874668972776192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827375.800074:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972776192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827375.800082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827375.800085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827375.800088:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972776192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827375.800093:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972776320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827375.800112:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827375.800139:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827375/real 1787827375] req@ffff9c1a49d8cf00 x1874668972776320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827375.800151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972776320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827375.800166:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827375.800170:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827375.800174:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972776320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827375.800181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972776448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827375.800202:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827375.800214:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827375/real 1787827375] req@ffff9c1a49d8da40 x1874668972776448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827375.800227:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972776448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827375.800235:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827375.800239:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827375.800242:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972776448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827375.800263:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972776576:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827375.800406:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827375.800420:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827375/real 1787827375] req@ffff9c1a49d8de00 x1874668972776576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827375.800434:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972776576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827375.800442:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827375.800446:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827375.800448:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972776576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827375.800455:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972776704:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827375.800475:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827375.800488:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827375/real 1787827375] req@ffff9c1a49d8c000 x1874668972776704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827375.800499:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972776704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827375.800507:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827375.800510:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827375.800514:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972776704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827376.311512:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827376.567658:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827377.335523:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827377.591551:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827378.359445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827378.615550:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827379.383440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827379.639552:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827380.407438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827380.663530:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827380.919661:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972776832:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787827380.919761:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827380.919811:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827380/real 1787827380] req@ffff9c1a49d8f480 x1874668972776832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827380.919841:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f480 x1874668972776832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827380.919869:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827380.919878:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827380.919887:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972776832:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827380.919932:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972776960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827380.919994:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827380.920018:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827380/real 1787827380] req@ffff9c1a49d8e940 x1874668972776960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827380.920035:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e940 x1874668972776960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827380.920045:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827380.920049:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827380.920053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972776960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827380.920092:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972777088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827380.920116:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827380.920130:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827380/real 1787827380] req@ffff9c1a49d8cb40 x1874668972777088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827380.920155:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972777088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827380.920171:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827380.920176:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827380.920181:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972777088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827380.920190:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972777216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827380.920229:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827380.920247:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827380/real 1787827380] req@ffff9c1a49d8ed00 x1874668972777216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827380.920267:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972777216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827380.920277:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827380.920281:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827380.920287:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972777216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827380.920295:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972777344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827380.920327:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827380.920340:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827380/real 1787827380] req@ffff9c1a49d8c3c0 x1874668972777344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827380.920400:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972777344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827380.920409:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827380.920413:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827380.920418:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972777344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827380.920441:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972777472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827380.920465:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827380.920477:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827380/real 1787827380] req@ffff9c1a49d8e580 x1874668972777472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827380.920490:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972777472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827380.920498:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827380.920501:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827380.920505:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972777472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827380.920512:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972777600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827380.920547:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827380.920560:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827380/real 1787827380] req@ffff9c1a49d8fc00 x1874668972777600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827380.920573:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972777600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827380.920582:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827380.920585:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827380.920588:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972777600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827381.431452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827381.687539:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827382.455420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827382.711574:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827383.479482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827383.735538:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827384.503465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827384.759561:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827385.527550:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827385.783481:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827386.039480:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827386.039489:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827386.039493:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827386.039501:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3836 00000100:00080000:0.0:1787827386.039510:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827386.039515:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827386.039547:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972777728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827386.039571:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827386.039574:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827386.039576:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827386.039579:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3836 00000100:00080000:0.0:1787827386.039581:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827386.039584:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827386.039592:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827386.039602:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788603c0 x1874668972777856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827386.039607:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972777728:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827386.039613:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827386.039617:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827386.039618:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827386.039621:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3836 00000100:00080000:0.0:1787827386.039622:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827386.039625:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827386.039628:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827386.039635:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972777984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827386.039643:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827386.039646:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827386.039647:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827386.039650:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3836 00000100:00080000:0.0:1787827386.039652:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827386.039654:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827386.039658:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827386.039665:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972778112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827386.039669:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827386.039678:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827386.039681:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827386.039683:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827386.039686:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3836 00000100:00080000:0.0:1787827386.039689:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827386.039692:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827386.039695:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827386.039711:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972778240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827386.039718:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827386.039723:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827386/real 1787827386] req@ffff9c1a788621c0 x1874668972777728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827386.039724:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827386.039727:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827386.039733:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3836 00000100:00080000:0.0:1787827386.039739:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827386.039745:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827386.039749:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827386.039754:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972777728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827386.039759:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972778368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827386.039787:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827386.039791:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827386.039794:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827386.039794:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827386.039804:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827386.039805:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3836 00000100:00080000:0.0:1787827386.039812:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827386.039815:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972777728:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827386.039822:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827386.039826:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827386.039833:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972778496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827386.039838:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972777856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827386.039867:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827386.039878:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827386/real 1787827386] req@ffff9c1a788603c0 x1874668972777856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827386.039887:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788603c0 x1874668972777856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827386.039892:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827386.039895:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827386.039897:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788603c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972777856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827386.039901:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972777984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827386.039912:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827386.039918:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827386/real 1787827386] req@ffff9c1a78862940 x1874668972777984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827386.039925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972777984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827386.039930:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827386.039932:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827386.039934:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972777984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827386.039938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972778112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827386.039947:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827386.039953:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827386/real 1787827386] req@ffff9c1a78860b40 x1874668972778112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827386.039961:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972778112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827386.039966:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827386.039968:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827386.039969:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972778112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827386.039973:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972778240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827386.039982:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827386.039988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827386/real 1787827386] req@ffff9c1a78861a40 x1874668972778240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827386.039995:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972778240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827386.040000:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827386.040002:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827386.040004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972778240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827386.040007:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972778368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827386.040017:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827386.040047:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827386/real 1787827386] req@ffff9c1a78860f00 x1874668972778368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827386.040058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972778368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827386.040066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827386.040069:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827386.040073:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972778368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827386.040096:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972778496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827386.040121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827386.040133:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827386/real 1787827386] req@ffff9c1a78862d00 x1874668972778496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827386.040146:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972778496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827386.040155:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827386.040158:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827386.040161:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972778496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827386.551433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827386.807516:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827387.575449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827387.831471:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827388.599424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827388.855461:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827389.623508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827389.879557:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827390.647440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827390.903579:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827391.159518:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827391.159529:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827391.159534:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827391.159545:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3841 00000100:00080000:1.0:1787827391.159558:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827391.159566:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827391.159608:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972778624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827391.159639:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827391.159657:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827391.159659:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827391.159663:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3841 00000100:00080000:1.0:1787827391.159672:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827391.159676:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827391.159686:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827391.159700:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972778752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827391.159711:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827391.159715:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827391.159716:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827391.159720:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3841 00000100:00080000:1.0:1787827391.159723:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827391.159726:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827391.159731:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827391.159740:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972778624:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827391.159744:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972778880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827391.159766:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827391.159769:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827391.159771:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827391.159800:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3841 00000100:00080000:1.0:1787827391.159806:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827391.159813:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827391.159821:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827391.159837:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972779008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827391.159849:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827391.159854:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827391.159856:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827391.159861:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3841 00000800:00000100:3.0:1787827391.159871:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827391.159873:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827391.159884:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827391.159890:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827391.159906:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972779136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827391.159918:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827391.159922:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827391.159925:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827391.159941:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3841 00000100:00080000:1.0:1787827391.159945:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827391.159950:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827391/real 1787827391] req@ffff9c1a78a7b840 x1874668972778624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827391.159954:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827391.159968:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827391.160014:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972779264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827391.160043:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972778624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827391.160049:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827391.160064:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827391.160069:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827391.160070:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827391.160082:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3841 00000100:00080000:3.0:1787827391.160088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827391.160090:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827391.160100:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827391.160108:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972778624:192.168.203.148@tcp:250:kworker.0 00000008:00100000:1.0:1787827391.160110:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827391.160128:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972779392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827391.160133:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972778752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827391.160181:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827391.160199:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827391/real 1787827391] req@ffff9c1a78a78000 x1874668972778752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827391.160212:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972778752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827391.160222:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827391.160225:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827391.160228:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972778752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827391.160253:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972778880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827391.160271:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827391.160280:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827391/real 1787827391] req@ffff9c1a78a7b480 x1874668972778880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827391.160293:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972778880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827391.160308:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827391.160311:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827391.160314:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972778880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827391.160320:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972779008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827391.160334:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827391.160343:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827391/real 1787827391] req@ffff9c1a78a792c0 x1874668972779008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827391.160354:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972779008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827391.160407:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827391.160411:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827391.160414:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972779008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827391.160421:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972779136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827391.160441:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827391.160452:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827391/real 1787827391] req@ffff9c1a78a78f00 x1874668972779136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827391.160464:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972779136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827391.160471:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827391.160474:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827391.160478:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972779136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827391.160484:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972779264:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827391.160507:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827391.160542:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827391/real 1787827391] req@ffff9c1a78a7a580 x1874668972779264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827391.160565:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972779264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827391.160582:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827391.160588:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827391.160595:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972779264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827391.160607:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972779392:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827391.160649:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827391.160664:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827391/real 1787827391] req@ffff9c1a78a79a40 x1874668972779392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827391.160686:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972779392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827391.160718:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827391.160724:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827391.160729:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972779392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827391.671592:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827391.927559:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827392.695451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827392.951519:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827393.719558:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827393.975621:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827394.743461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827394.999529:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827395.767483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827396.023588:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827396.279571:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972779520:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787827396.279638:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827396.279684:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827396/real 1787827396] req@ffff9c1a49d8e1c0 x1874668972779520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827396.279703:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972779520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827396.279717:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827396.279723:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827396.279728:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972779520:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827396.279751:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972779648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827396.279781:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827396.279797:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827396/real 1787827396] req@ffff9c1a49d8fc00 x1874668972779648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827396.279817:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972779648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827396.279834:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827396.279841:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827396.279849:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972779648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827396.279859:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972779776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827396.279874:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827396.279883:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827396/real 1787827396] req@ffff9c1a49d8e580 x1874668972779776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827396.279896:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972779776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827396.279901:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827396.279903:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827396.279905:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972779776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827396.279908:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972779904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827396.279919:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827396.279925:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827396/real 1787827396] req@ffff9c1a49d8c3c0 x1874668972779904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827396.279932:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972779904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827396.279937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827396.279939:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827396.279941:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972779904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827396.279944:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972780032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827396.279969:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827396.279979:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827396/real 1787827396] req@ffff9c1a49d8ed00 x1874668972780032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827396.279987:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972780032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827396.279997:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827396.279999:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827396.280001:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972780032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827396.280014:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972780160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827396.280027:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827396.280034:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827396/real 1787827396] req@ffff9c1a49d8cb40 x1874668972780160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827396.280041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972780160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827396.280046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827396.280048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827396.280050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972780160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827396.280054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972780288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827396.280063:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827396.280068:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827396/real 1787827396] req@ffff9c1a49d8e940 x1874668972780288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827396.280075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e940 x1874668972780288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827396.280080:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827396.280082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827396.280084:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972780288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827396.791524:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827397.047525:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827397.815509:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827398.071551:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827398.839489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827399.095489:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827399.863473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827400.119766:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827400.887549:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827401.143709:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827401.399681:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972780416:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787827401.399804:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827401.399880:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827401/real 1787827401] req@ffff9c1a49d8f480 x1874668972780416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827401.399905:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f480 x1874668972780416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827401.399940:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827401.399949:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827401.399960:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972780416:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827401.400002:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972780544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827401.400031:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827401.400051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827401/real 1787827401] req@ffff9c1a49d8c000 x1874668972780544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827401.400084:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972780544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827401.400100:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827401.400109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827401.400121:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972780544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827401.400138:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972780672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827401.400175:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827401.400190:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827401/real 1787827401] req@ffff9c1a49d8de00 x1874668972780672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827401.400207:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972780672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827401.400218:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827401.400222:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827401.400227:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972780672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827401.400235:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972780800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827401.400260:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827401.400271:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827401/real 1787827401] req@ffff9c1a49d8da40 x1874668972780800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827401.400281:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972780800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827401.400289:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827401.400292:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827401.400295:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972780800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827401.400300:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972780928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827401.400315:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827401.400322:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827401/real 1787827401] req@ffff9c1a49d8cf00 x1874668972780928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827401.400344:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972780928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827401.400376:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827401.400379:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827401.400381:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972780928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827401.400402:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972781056:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827401.400420:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827401.400429:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827401/real 1787827401] req@ffff9c1a49d8f0c0 x1874668972781056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827401.400439:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972781056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827401.400446:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827401.400448:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827401.400451:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972781056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827401.400455:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972781184:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827401.400467:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827401.400474:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827401/real 1787827401] req@ffff9c1a49d8d680 x1874668972781184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827401.400501:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972781184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827401.400514:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827401.400517:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827401.400520:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972781184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827401.911446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827402.167584:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827402.935466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827403.191646:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827403.959478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827404.215556:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827404.983467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827405.239484:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827406.007432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827406.263506:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827406.519607:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972781312:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787827406.519673:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827406.519724:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827406/real 1787827406] req@ffff9c1a49d8c780 x1874668972781312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827406.519753:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972781312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827406.519781:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827406.519789:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827406.519795:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972781312:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827406.519822:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972781440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827406.519853:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827406.519863:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827406/real 1787827406] req@ffff9c1a49d8d680 x1874668972781440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827406.519876:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972781440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827406.519882:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827406.519899:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827406.519901:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972781440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827406.519906:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972781568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827406.519921:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827406.519938:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827406/real 1787827406] req@ffff9c1a49d8f0c0 x1874668972781568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827406.519946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972781568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827406.519951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827406.519953:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827406.519955:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972781568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827406.519959:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972781696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827406.519969:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827406.519974:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827406/real 1787827406] req@ffff9c1a49d8cf00 x1874668972781696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827406.519981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972781696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827406.519991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827406.519993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827406.519996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972781696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827406.519999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972781824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827406.520008:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827406.520014:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827406/real 1787827406] req@ffff9c1a49d8da40 x1874668972781824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827406.520020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972781824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827406.520025:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827406.520027:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827406.520029:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972781824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827406.520032:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972781952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827406.520042:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827406.520047:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827406/real 1787827406] req@ffff9c1a49d8de00 x1874668972781952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827406.520054:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972781952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827406.520059:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827406.520060:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827406.520063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972781952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827406.520066:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972782080:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827406.520075:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827406.520080:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827406/real 1787827406] req@ffff9c1a49d8c000 x1874668972782080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827406.520087:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972782080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827406.520092:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827406.520094:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827406.520095:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972782080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827407.031452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827407.287490:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827408.055415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827408.311504:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827409.079458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827409.335620:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827410.103458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827410.359554:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827411.127450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827411.383593:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827411.639535:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827411.639548:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827411.639581:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827411.639600:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3861 00000100:00080000:1.0:1787827411.639612:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827411.639619:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827411.639673:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78b40 x1874668972782208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827411.639716:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827411.639722:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827411.639726:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827411.639734:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3861 00000100:00080000:1.0:1787827411.639741:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827411.639746:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827411.639758:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827411.639801:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972782336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827411.639814:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972782208:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827411.639823:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827411.639830:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827411.639833:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827411.639839:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3861 00000100:00080000:1.0:1787827411.639845:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827411.639872:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827411.639880:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827411.639908:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972782464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827411.639923:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827411.639928:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827411.639930:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827411.639940:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827411.639941:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3861 00000100:00080000:1.0:1787827411.639953:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827411.639958:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827411.639983:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827411.640002:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972782592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827411.640017:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827411.640025:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827411/real 1787827411] req@ffff9c1a78a78b40 x1874668972782208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827411.640026:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827411.640033:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827411.640049:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3861 00000100:00080000:1.0:1787827411.640066:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827411.640101:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827411.640109:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78b40 x1874668972782208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827411.640115:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787827411.640135:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827411.640143:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827411.640151:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972782720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827411.640155:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972782208:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827411.640182:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827411.640187:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827411.640190:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827411.640215:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972782336:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827411.640216:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3861 00000100:00080000:1.0:1787827411.640241:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827411.640260:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787827411.640267:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:1.0:1787827411.640270:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827411.640285:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827411/real 1787827411] req@ffff9c1a78a79a40 x1874668972782336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827411.640298:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972782848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827411.640307:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972782336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827411.640349:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827411.640352:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827411.640357:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972782336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827411.640365:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972782464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827411.640383:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827411.640428:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827411.640434:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827411.640437:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827411.640446:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3861 00000100:00080000:1.0:1787827411.640455:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827411.640461:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827411.640472:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827411.640475:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827411/real 1787827411] req@ffff9c1a78a7a580 x1874668972782464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827411.640501:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972782976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827411.640507:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972782464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827411.640568:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827411.640576:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827411.640584:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972782464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827411.640618:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972782592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827411.640653:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827411.640672:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827411/real 1787827411] req@ffff9c1a78a78f00 x1874668972782592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827411.640693:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972782592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827411.640702:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827411.640705:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827411.640708:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972782592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827411.640714:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972782720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827411.640737:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827411.640748:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827411/real 1787827411] req@ffff9c1a78a792c0 x1874668972782720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827411.640759:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972782720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827411.640767:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827411.640770:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827411.640773:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972782720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827411.640801:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972782848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827411.640841:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827411.640895:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827411/real 1787827411] req@ffff9c1a78a7b480 x1874668972782848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827411.640923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972782848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827411.640932:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827411.640952:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827411.640956:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972782848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827411.640964:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972782976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827411.640990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827411.641002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827411/real 1787827411] req@ffff9c1a78a78000 x1874668972782976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827411.641016:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972782976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827411.641026:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827411.641029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827411.641033:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972782976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827412.151435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827412.407587:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827413.175499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827413.431698:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827414.199475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827414.455592:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827415.223426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827415.479572:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827416.247408:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827416.503527:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827416.759448:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827416.759459:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827416.759464:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827416.759474:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3867 00000100:00080000:1.0:1787827416.759485:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827416.759492:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827416.759528:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972783104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827416.759560:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827416.759564:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827416.759566:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827416.759570:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3867 00000100:00080000:1.0:1787827416.759574:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827416.759577:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827416.759586:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827416.759604:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972783232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827416.759630:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827416.759652:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827416.759667:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827416.759674:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3867 00000100:00080000:1.0:1787827416.759679:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827416.759683:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827416.759688:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827416.759700:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972783360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827416.759710:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827416.759713:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827416.759715:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827416.759718:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3867 00000100:00080000:1.0:1787827416.759721:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827416.759724:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827416.759728:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827416.759738:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972783488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827416.759746:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827416.759749:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827416.759751:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827416.759753:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3867 00000100:00080000:1.0:1787827416.759756:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827416.759775:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827416.759778:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827416.759788:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972783616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827416.759794:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827416.759796:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827416.759797:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827416.759801:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3867 00000100:00080000:1.0:1787827416.759804:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827416.759807:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827416.759810:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827416.759833:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972783744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827416.759841:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827416.759850:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827416.759853:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827416.759870:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3867 00000100:00080000:1.0:1787827416.759873:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827416.759877:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827416.759882:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827416.759898:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972783872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827417.271470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827417.527676:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827418.295466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827418.551557:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827419.319456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827419.575583:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827420.343499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827420.599631:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827421.367535:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827421.623486:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827421.879578:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972784000:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787827421.879637:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827421.879675:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827421/real 1787827421] req@ffff9c1a49d8f480 x1874668972784000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827421.879695:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f480 x1874668972784000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827421.879705:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827421.879711:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827421.879716:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972784000:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827421.879742:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972784128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827421.879766:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827421.879799:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827421/real 1787827421] req@ffff9c1a49d8e940 x1874668972784128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827421.879837:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e940 x1874668972784128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827421.879860:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827421.879865:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827421.879868:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972784128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827421.879874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972784256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827421.879892:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827421.879900:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827421/real 1787827421] req@ffff9c1a49d8cb40 x1874668972784256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827421.879908:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972784256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827421.879913:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827421.879915:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827421.879917:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972784256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827421.879921:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972784384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827421.879930:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827421.879936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827421/real 1787827421] req@ffff9c1a49d8ed00 x1874668972784384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827421.879943:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972784384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827421.879948:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827421.879950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827421.879952:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972784384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827421.879964:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972784512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827421.879974:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827421.879980:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827421/real 1787827421] req@ffff9c1a49d8c3c0 x1874668972784512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827421.879992:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972784512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827421.879997:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827421.879999:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827421.880001:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972784512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827421.880004:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972784640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827421.880014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827421.880019:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827421/real 1787827421] req@ffff9c1a49d8e580 x1874668972784640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827421.880027:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972784640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827421.880031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827421.880033:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827421.880035:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972784640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827421.880038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972784768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827421.880047:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827421.880053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827421/real 1787827421] req@ffff9c1a49d8fc00 x1874668972784768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827421.880059:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972784768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827421.880068:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827421.880070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827421.880072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972784768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827422.391450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827422.647544:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827423.415435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827423.671528:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827424.439477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827424.695636:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827425.463475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827425.719668:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827426.487467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827426.743551:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827426.999525:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827426.999538:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827426.999543:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827426.999556:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3877 00000100:00080000:0.0:1787827426.999571:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827426.999579:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827426.999632:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972784896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827426.999672:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827426.999676:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827426.999678:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827426.999682:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3877 00000100:00080000:0.0:1787827426.999685:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827426.999689:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827426.999701:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827426.999713:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972785024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827426.999723:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827426.999728:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827426.999730:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827426.999734:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3877 00000100:00080000:0.0:1787827426.999737:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827426.999757:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827426.999763:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827426.999779:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972785152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827426.999791:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972784896:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827426.999795:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827426.999806:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827426.999815:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827426.999819:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3877 00000100:00080000:0.0:1787827426.999823:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827426.999829:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827426.999834:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827426.999846:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972785280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827426.999855:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827426.999859:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827426.999862:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827426.999865:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3877 00000100:00080000:0.0:1787827426.999868:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827426.999871:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827426.999877:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827426.999887:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972785408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827426.999902:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827426.999905:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827426.999910:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827426.999913:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827426.999918:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3877 00000100:00080000:0.0:1787827426.999922:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827426.999926:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827426.999932:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827426.999944:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972785536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827426.999952:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827426.999956:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827426.999958:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827426.999962:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3877 00000100:00080000:0.0:1787827426.999970:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827426.999971:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827426/real 1787827426] req@ffff9c1a78863840 x1874668972784896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827426.999978:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827426.999984:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827427.000003:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972785664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827427.000022:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972784896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827427.000039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827427.000046:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827427.000053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972784896:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827427.000066:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972785024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827427.000097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827427.000112:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827426/real 1787827427] req@ffff9c1a788612c0 x1874668972785024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827427.000136:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972785024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827427.000147:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827427.000151:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827427.000154:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972785024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827427.000178:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972785152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827427.000198:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827427.000207:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827426/real 1787827427] req@ffff9c1a78862d00 x1874668972785152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827427.000217:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972785152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827427.000223:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827427.000226:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827427.000228:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972785152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827427.000233:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972785280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827427.000245:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827427.000252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827426/real 1787827427] req@ffff9c1a78860f00 x1874668972785280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827427.000261:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972785280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827427.000273:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827427.000275:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827427.000278:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972785280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827427.000282:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972785408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827427.000293:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827427.000300:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827426/real 1787827427] req@ffff9c1a78861a40 x1874668972785408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827427.000309:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972785408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827427.000315:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827427.000317:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827427.000320:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972785408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827427.000324:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972785536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827427.000336:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827427.000343:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827426/real 1787827427] req@ffff9c1a78860b40 x1874668972785536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827427.000385:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972785536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827427.000392:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827427.000394:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827427.000397:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972785536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827427.000403:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972785664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827427.000418:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827427.000427:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827427/real 1787827427] req@ffff9c1a78862940 x1874668972785664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827427.000436:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972785664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827427.000442:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827427.000444:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827427.000447:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972785664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827427.511439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827427.767556:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827428.535486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827428.791626:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827429.559433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827429.815532:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827430.583490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827430.839609:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827431.607418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827431.863530:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827432.119526:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827432.119544:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827432.119549:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827432.119562:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3882 00000100:00080000:1.0:1787827432.119574:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827432.119583:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827432.119626:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972785792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827432.119652:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827432.119656:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827432.119658:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827432.119666:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3882 00000100:00080000:1.0:1787827432.119670:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827432.119674:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827432.119683:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827432.119693:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972785920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827432.119703:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827432.119725:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827432.119727:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827432.119731:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3882 00000100:00080000:1.0:1787827432.119734:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827432.119738:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827432.119743:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827432.119772:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972786048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827432.119780:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827432.119783:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827432.119785:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827432.119788:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3882 00000100:00080000:1.0:1787827432.119792:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827432.119795:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827432.119798:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827432.119809:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972786176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827432.119819:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972785792:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827432.119820:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827432.119835:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827432.119838:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827432.119843:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3882 00000100:00080000:1.0:1787827432.119845:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827432.119849:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827432.119854:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827432.119886:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972786304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827432.119898:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827432.119902:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827432.119905:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827432.119911:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3882 00000100:00080000:1.0:1787827432.119917:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827432.119923:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827432.119931:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827432.119947:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972786432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827432.119959:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827432.119963:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827432.119964:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827432.119968:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3882 00000100:00080000:1.0:1787827432.119972:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827432.119982:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827432.119990:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827432.119998:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827432.120012:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972786560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827432.120086:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827432/real 1787827432] req@ffff9c1a78a79e00 x1874668972785792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827432.120114:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972785792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827432.120129:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827432.120135:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827432.120141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972785792:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827432.120153:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972785920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827432.120186:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827432.120202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827432/real 1787827432] req@ffff9c1a78a7b0c0 x1874668972785920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827432.120217:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972785920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827432.120228:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827432.120232:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827432.120237:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972785920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827432.120244:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972786048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827432.120263:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827432.120275:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827432/real 1787827432] req@ffff9c1a78a7bc00 x1874668972786048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827432.120288:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972786048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827432.120299:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827432.120302:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827432.120306:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972786048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827432.120342:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972786176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827432.120425:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827432.120440:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827432/real 1787827432] req@ffff9c1a78a78780 x1874668972786176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827432.120456:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972786176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827432.120467:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827432.120471:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827432.120475:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972786176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827432.120483:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972786304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827432.120502:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827432.120513:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827432/real 1787827432] req@ffff9c1a78a7a1c0 x1874668972786304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827432.120527:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972786304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827432.120541:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827432.120546:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827432.120553:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972786304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827432.120565:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972786432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827432.120604:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827432.120627:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827432/real 1787827432] req@ffff9c1a78a783c0 x1874668972786432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827432.120662:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972786432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827432.120673:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827432.120677:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827432.120681:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972786432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827432.120689:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972786560:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827432.120744:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827432.120761:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827432/real 1787827432] req@ffff9c1a78a7a940 x1874668972786560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827432.120777:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972786560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827432.120788:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827432.120792:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827432.120796:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972786560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827432.631451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827432.887609:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827433.655463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827433.911481:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827434.679467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827434.935545:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827435.703488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827435.959566:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827436.727454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827436.727473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827436.983617:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827437.239478:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827437.239488:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827437.239493:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827437.239503:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3887 00000100:00080000:1.0:1787827437.239515:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827437.239522:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827437.239560:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972786688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827437.239592:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827437.239596:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827437.239599:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827437.239603:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3887 00000100:00080000:1.0:1787827437.239607:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827437.239611:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827437.239621:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827437.239636:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972786816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827437.239648:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827437.239654:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827437.239654:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972786688:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827437.239658:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827437.239667:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3887 00000100:00080000:1.0:1787827437.239686:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827437.239691:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827437.239696:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827437.239711:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972786944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827437.239733:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827437.239737:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827437.239739:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827437.239743:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3887 00000100:00080000:1.0:1787827437.239747:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827437.239752:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827437.239757:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827437.239769:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972787072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827437.239780:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827437.239784:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827437.239787:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827437.239793:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3887 00000100:00080000:1.0:1787827437.239797:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827437.239800:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972786816:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827437.239805:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827437.239815:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827437.239830:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972787200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827437.239843:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827437.239845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972786944:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827437.239848:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827437.239853:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827437.239860:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3887 00000100:00080000:1.0:1787827437.239866:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827437.239871:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827437.239878:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827437.239883:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972787072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827437.239894:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972787328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827437.239909:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827437.239913:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827437.239915:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827437.239920:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3887 00000100:00080000:1.0:1787827437.239924:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827437.239929:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827437.239936:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827437.239937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972787200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827437.239956:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972787456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827437.239966:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972787328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827437.239985:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972787456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827437.751475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827437.751475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827438.007464:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827438.775429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827438.775429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827439.031529:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827439.799464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827439.799464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827440.055514:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827440.311896:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827437/real 1787827440] req@ffff9c1a78a78f00 x1874668972787200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827440.311925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972787200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827440.311937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827440.311941:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827440.311945:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972787200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827440.311952:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827437/real 1787827440] req@ffff9c1a78a7a580 x1874668972787328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827440.311963:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972787328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827440.311971:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827440.311975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827440.311978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972787328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827440.311983:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827437/real 1787827440] req@ffff9c1a78a78000 x1874668972786816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827440.311995:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972786816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827440.312002:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827440.312006:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827440.312008:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972786816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827440.312015:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827437/real 1787827440] req@ffff9c1a78a7b480 x1874668972786944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827440.312026:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972786944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827440.312033:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827440.312036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827440.312039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972786944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827440.312046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827437/real 1787827440] req@ffff9c1a78a792c0 x1874668972787072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827440.312055:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972787072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827440.312062:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827440.312066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827440.312069:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972787072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827440.312075:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827437/real 1787827440] req@ffff9c1a78a7b840 x1874668972786688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827440.312085:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972786688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827440.312091:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827440.312094:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827440.312098:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972786688:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827440.312127:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827437/real 1787827440] req@ffff9c1a78a79a40 x1874668972787456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827440.312137:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972787456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827440.312144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827440.312147:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827440.312150:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972787456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827440.823433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827441.079529:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827441.847452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827442.103568:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827442.359817:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972787584:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787827442.359979:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827442.360112:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827442/real 1787827442] req@ffff9c1a49d8e1c0 x1874668972787584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827442.360175:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972787584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827442.360197:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827442.360206:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827442.360213:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972787584:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827442.360229:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972787712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827442.360279:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827442.360309:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827442/real 1787827442] req@ffff9c1a49d8fc00 x1874668972787712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827442.360534:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972787712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827442.360557:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827442.360566:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827442.360573:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972787712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827442.360587:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972787840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827442.360658:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827442.360687:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827442/real 1787827442] req@ffff9c1a49d8e580 x1874668972787840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827442.360707:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972787840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827442.360726:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827442.360731:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827442.360735:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972787840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827442.360743:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972787968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827442.360773:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827442.360791:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827442/real 1787827442] req@ffff9c1a49d8c3c0 x1874668972787968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827442.360814:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972787968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827442.360833:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827442.360839:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827442.360845:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972787968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827442.360905:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972788096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827442.360947:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827442.360970:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827442/real 1787827442] req@ffff9c1a49d8ed00 x1874668972788096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827442.360993:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972788096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827442.361010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827442.361017:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827442.361024:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972788096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827442.361036:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972788224:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827442.361075:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827442.361097:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827442/real 1787827442] req@ffff9c1a49d8cb40 x1874668972788224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827442.361134:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972788224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827442.361151:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827442.361158:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827442.361164:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972788224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827442.361175:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972788352:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827442.361212:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827442.361259:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827442/real 1787827442] req@ffff9c1a49d8e940 x1874668972788352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827442.361284:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e940 x1874668972788352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827442.361300:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827442.361306:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827442.361313:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972788352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827442.871444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827443.127612:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827443.895474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827444.151668:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827444.919442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827445.175604:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827445.943456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827446.199548:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827446.967420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827447.223593:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827447.479619:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972788480:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787827447.479715:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827447.479768:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827447/real 1787827447] req@ffff9c1a49d8f480 x1874668972788480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827447.479807:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f480 x1874668972788480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827447.479835:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827447.479858:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827447.479863:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972788480:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827447.479872:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972788608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827447.479893:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827447.479902:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827447/real 1787827447] req@ffff9c1a49d8c000 x1874668972788608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827447.479912:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972788608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827447.479935:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827447.479938:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827447.479940:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972788608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827447.479948:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972788736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827447.479963:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827447.479971:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827447/real 1787827447] req@ffff9c1a49d8de00 x1874668972788736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827447.479980:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972788736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827447.480005:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827447.480023:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827447.480027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972788736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827447.480076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972788864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827447.480093:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827447.480103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827447/real 1787827447] req@ffff9c1a49d8da40 x1874668972788864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827447.480113:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972788864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827447.480120:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827447.480138:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827447.480141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972788864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827447.480145:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972788992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827447.480157:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827447.480164:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827447/real 1787827447] req@ffff9c1a49d8cf00 x1874668972788992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827447.480172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972788992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827447.480178:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827447.480180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827447.480182:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972788992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827447.480186:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972789120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827447.480197:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827447.480203:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827447/real 1787827447] req@ffff9c1a49d8f0c0 x1874668972789120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827447.480211:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972789120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827447.480216:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827447.480218:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827447.480220:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972789120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827447.480224:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972789248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827447.480234:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827447.480240:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827447/real 1787827447] req@ffff9c1a49d8d680 x1874668972789248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827447.480248:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972789248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827447.480258:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827447.480260:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827447.480262:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972789248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827447.991459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827448.247508:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827449.015415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827449.271542:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827450.039450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827450.295527:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827451.063428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827451.319487:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827452.087457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827452.343455:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827452.599655:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972789376:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787827452.599772:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827452.599852:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827452/real 1787827452] req@ffff9c1a49d8c780 x1874668972789376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827452.599911:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972789376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827452.599932:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827452.599945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827452.599954:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972789376:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827452.599989:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972789504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827452.600021:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827452.600038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827452/real 1787827452] req@ffff9c1a49d8d680 x1874668972789504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827452.600048:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972789504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827452.600057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827452.600062:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827452.600071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972789504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827452.600093:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972789632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827452.600125:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827452.600142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827452/real 1787827452] req@ffff9c1a49d8f0c0 x1874668972789632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827452.600158:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972789632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827452.600176:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827452.600181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827452.600188:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972789632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827452.600199:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972789760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827452.600226:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827452.600238:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827452/real 1787827452] req@ffff9c1a49d8cf00 x1874668972789760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827452.600246:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972789760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827452.600252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827452.600254:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827452.600256:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972789760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827452.600261:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972789888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827452.600274:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827452.600280:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827452/real 1787827452] req@ffff9c1a49d8da40 x1874668972789888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827452.600287:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972789888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827452.600292:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827452.600294:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827452.600297:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972789888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827452.600301:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972790016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827452.600313:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827452.600319:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827452/real 1787827452] req@ffff9c1a49d8de00 x1874668972790016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827452.600326:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972790016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827452.600331:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827452.600333:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827452.600335:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972790016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827452.600350:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972790144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827452.600367:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827452.600415:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827452/real 1787827452] req@ffff9c1a49d8c000 x1874668972790144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827452.600426:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972790144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827452.600435:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827452.600439:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827452.600442:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972790144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827453.111474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827453.367478:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827454.135453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827454.391585:0:15688:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 27 locks of 27 total in blp. (1 blwis in pool) 00010000:00010000:0.0:1787827454.391630:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000eeff973a/0x531a24091f987351 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bae2e3 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.391665:0:15103:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 27 locks of 23 total in blp. (1 blwis in pool) 00000080:00200000:0.0:1787827454.391669:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x163:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.391684:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fd400/0x531a24091f987319 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bae257 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.391690:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x163:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:3.0:1787827454.391714:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827454.391735:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000bd5fa4e2/0x531a24091f987343 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bae2c0 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.391749:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x65:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.391750:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fd600/0x531a24091f98730b lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bae23b expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827454.391777:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x65:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.391854:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fda00/0x531a24091f987304 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bae22d expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.391857:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000e3e3fa4d/0x531a24091f987335 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x2b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bae29d expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.391939:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x63:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.391949:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x63:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827454.391964:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000002719a2a1/0x531a24091f987327 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x10:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bae27a expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.391975:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fcc00/0x531a24091f9872ef lrc: 2/0,0 mode: PR/PR res: [0x240000401:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bae1f5 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827454.392019:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x2f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.392046:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x2f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.392066:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25c00/0x531a24091f98728d lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x28:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bae0f2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.392069:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000ecd06b33/0x531a24091f9872cc lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bae19a expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.392120:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x7b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.392126:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x7b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.392146:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24e00/0x531a24091f98727f lrc: 2/0,0 mode: PR/PR res: [0x240000400:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bae0d6 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.392163:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000effe73fa/0x531a24091f9872be lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x2e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bae177 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.392198:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25800/0x531a24091f987278 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x28:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bae0c8 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827454.392223:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000403:0x25:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.392235:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x25:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.392247:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25a00/0x531a24091f98724e lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bae058 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.392253:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000d351dd5a/0x531a24091f9872b0 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bae154 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.392273:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25200/0x531a24091f987240 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bae03c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827454.392280:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.392298:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.392307:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24a00/0x531a24091f987239 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bae02e expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.392313:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000da0cc5fe/0x531a24091f98726a lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bae09e expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.392383:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.392394:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827454.392408:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000aaf421f1/0x531a24091f98725c lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x38:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bae07b expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.392423:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x82:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.392428:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x82:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.392443:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4a00/0x531a24091f987208 lrc: 2/0,0 mode: PR/PR res: [0x240000401:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badfb0 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827454.392454:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000005f53f123/0x531a24091f987167 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x13:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bade21 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.392466:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x35:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.392466:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4c00/0x531a24091f9871ec lrc: 2/0,0 mode: PR/PR res: [0x240000401:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badf6a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827454.392479:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x35:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.392497:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4400/0x531a24091f9871d7 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x1c:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badf32 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.392515:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000009a1cc7f6/0x531a24091f987159 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753baddfe expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.392535:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x5b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.392541:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x5b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.392548:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5200/0x531a24091f9871c9 lrc: 2/0,0 mode: PR/PR res: [0x240000400:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badf16 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.392564:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000014246f40/0x531a24091f98714b lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753badddb expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.392641:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000403:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.392649:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5600/0x531a24091f9871c2 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x1c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badf08 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827454.392651:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827454.392677:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000087ec420c/0x531a24091f987105 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753badd25 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.392696:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000403:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.392704:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2200/0x531a24091f98719f lrc: 2/0,0 mode: PR/PR res: [0x240000401:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badead expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827454.392705:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000403:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827454.392753:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000b667ce01/0x531a24091f9870f7 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x2a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753badd02 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.392778:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2a00/0x531a24091f98718a lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bade75 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.392795:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x61:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.392811:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2400/0x531a24091f98717c lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bade59 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827454.392821:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x61:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.392855:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb3600/0x531a24091f987175 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x14:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bade4b expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.392880:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000bf78db19/0x531a24091f9870e2 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753badcd8 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.392931:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478800/0x531a24091f987087 lrc: 2/0,0 mode: PR/PR res: [0x240000401:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badbf8 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827454.392933:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.392956:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.392963:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478a00/0x531a24091f987025 lrc: 2/0,0 mode: PR/PR res: [0x240000401:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badafc expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827454.393000:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000078f89158/0x531a24091f9870c6 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753badc92 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.393006:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007e00/0x531a24091f987009 lrc: 2/0,0 mode: PR/PR res: [0x240000404:0x6b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badab6 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827454.393043:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.393062:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.393063:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007200/0x531a24091f986fbc lrc: 2/0,0 mode: PR/PR res: [0x240000401:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad9eb expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.393079:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000b0ab7367/0x531a24091f987064 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x20:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753badb96 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.393106:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf600/0x531a24091f986f8b lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad96d expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.393119:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x4f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.393134:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbee00/0x531a24091f986f7d lrc: 2/0,0 mode: PR/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad951 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827454.393135:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x4f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827454.393150:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000ec26f951/0x531a24091f986fed lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bada62 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.393202:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x59:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.393213:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x59:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.393219:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf200/0x531a24091f986f76 lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad943 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.393228:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000001fcc4b57/0x531a24091f986fdf lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bada3f expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.393268:0:15103:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1800/0x531a24091f986e9d lrc: 2/0,0 mode: PR/PR res: [0x240000401:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad71a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827454.393276:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x5d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.393284:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x5d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827454.393311:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000c1a95828/0x531a24091f986f99 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bad990 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.393323:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fd400/0x531a24091f987319 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x13:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae257 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.393326:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x5f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.393339:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x5f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.393362:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fd600/0x531a24091f98730b lrc: 1/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae23b expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.393368:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000dba61ca0/0x531a24091f986f68 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x2d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bad919 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.393418:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fda00/0x531a24091f987304 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x13:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae22d expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827454.393422:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x67:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.393429:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fcc00/0x531a24091f9872ef lrc: 1/0,0 mode: --/PR res: [0x240000401:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae1f5 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827454.393432:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x67:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827454.393450:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000c5186ada/0x531a24091f986f5a lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bad8f6 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.393457:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25c00/0x531a24091f98728d lrc: 1/0,0 mode: --/PR res: [0x240000402:0x28:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae0f2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.393504:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x85:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.393509:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24e00/0x531a24091f98727f lrc: 1/0,0 mode: --/PR res: [0x240000400:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae0d6 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827454.393512:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x85:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.393517:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25800/0x531a24091f987278 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x28:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae0c8 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.393524:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000059be989b/0x531a24091f986f3e lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x16:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bad8b0 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.393546:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25a00/0x531a24091f98724e lrc: 1/0,0 mode: --/PR res: [0x240000402:0x4:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae058 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.393570:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x3b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.393586:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x3b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.393588:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25200/0x531a24091f987240 lrc: 1/0,0 mode: --/PR res: [0x240000400:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae03c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827454.393621:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24a00/0x531a24091f987239 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x4:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae02e expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827454.393632:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4a00/0x531a24091f987208 lrc: 1/0,0 mode: --/PR res: [0x240000401:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badfb0 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827454.393639:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000003578f3f5/0x531a24091f986f30 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bad88d expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.393647:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4c00/0x531a24091f9871ec lrc: 1/0,0 mode: --/PR res: [0x240000401:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf6a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827454.393740:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4400/0x531a24091f9871d7 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x1c:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf32 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.393746:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x78:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.393767:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x78:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.393768:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5200/0x531a24091f9871c9 lrc: 1/0,0 mode: --/PR res: [0x240000400:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf16 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.393780:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000017bcc6f2/0x531a24091f986a4b lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x473:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bac53f expref: -99 pid: 71124 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827454.393809:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5600/0x531a24091f9871c2 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x1c:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf08 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1787827454.393812:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x16a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1787827454.393830:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x16a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:0.0:1787827454.393852:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000aadaddaf/0x531a24091f986ed5 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x472:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bad798 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.393865:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2200/0x531a24091f98719f lrc: 1/0,0 mode: --/PR res: [0x240000401:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badead expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827454.393910:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x168:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.393912:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2a00/0x531a24091f98718a lrc: 1/0,0 mode: --/PR res: [0x240000402:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade75 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1787827454.393940:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x168:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.393943:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2400/0x531a24091f98717c lrc: 1/0,0 mode: --/PR res: [0x240000400:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade59 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.393960:0:15688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000d8c8a625/0x531a24091f986eb9 lrc: 2/0,0 mode: PR/PR res: [0x2c0000401:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0x9505c05753bad752 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.393969:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb3600/0x531a24091f987175 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x14:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade4b expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827454.393997:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478800/0x531a24091f987087 lrc: 1/0,0 mode: --/PR res: [0x240000401:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badbf8 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827454.394010:0:15688:0:(vvp_io.c:1885:vvp_io_init()) [0x200000402:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00010000:00010000:1.0:1787827454.394054:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478a00/0x531a24091f987025 lrc: 1/0,0 mode: --/PR res: [0x240000401:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badafc expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1787827454.394056:0:15688:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:1.0:1787827454.394103:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007e00/0x531a24091f987009 lrc: 1/0,0 mode: --/PR res: [0x240000404:0x6b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badab6 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827454.394114:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007200/0x531a24091f986fbc lrc: 1/0,0 mode: --/PR res: [0x240000401:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad9eb expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827454.394123:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf600/0x531a24091f986f8b lrc: 1/0,0 mode: --/PR res: [0x240000402:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad96d expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.394144:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000eeff973a/0x531a24091f987351 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae2e3 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394173:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbee00/0x531a24091f986f7d lrc: 1/0,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad951 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827454.394224:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf200/0x531a24091f986f76 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x6:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad943 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.394225:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000bd5fa4e2/0x531a24091f987343 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae2c0 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394238:0:15103:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1800/0x531a24091f986e9d lrc: 1/0,0 mode: --/PR res: [0x240000401:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad71a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827454.394257:0:15103:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 27 locks packed 00010000:00010000:0.0:1787827454.394284:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000e3e3fa4d/0x531a24091f987335 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x2b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae29d expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394288:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fd400/0x531a24091f987319 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x13:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae257 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.394339:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000002719a2a1/0x531a24091f987327 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x10:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae27a expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394353:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fd600/0x531a24091f98730b lrc: 0/0,0 mode: --/PR res: [0x240000400:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae23b expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.394362:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000ecd06b33/0x531a24091f9872cc lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae19a expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.394418:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000effe73fa/0x531a24091f9872be lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x2e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae177 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.394427:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000d351dd5a/0x531a24091f9872b0 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae154 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394429:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fda00/0x531a24091f987304 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x13:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae22d expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.394447:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000da0cc5fe/0x531a24091f98726a lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae09e expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394458:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fcc00/0x531a24091f9872ef lrc: 0/0,0 mode: --/PR res: [0x240000401:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae1f5 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827454.394514:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000aaf421f1/0x531a24091f98725c lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x38:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae07b expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394600:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25c00/0x531a24091f98728d lrc: 0/0,0 mode: --/PR res: [0x240000402:0x28:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae0f2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.394603:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000005f53f123/0x531a24091f987167 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x13:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bade21 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394619:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24e00/0x531a24091f98727f lrc: 0/0,0 mode: --/PR res: [0x240000400:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae0d6 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.394655:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000009a1cc7f6/0x531a24091f987159 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753baddfe expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.394702:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000014246f40/0x531a24091f98714b lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badddb expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394703:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25800/0x531a24091f987278 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x28:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae0c8 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.394757:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000087ec420c/0x531a24091f987105 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badd25 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394767:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25a00/0x531a24091f98724e lrc: 0/0,0 mode: --/PR res: [0x240000402:0x4:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae058 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.394768:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000b667ce01/0x531a24091f9870f7 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x2a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badd02 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.394853:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000bf78db19/0x531a24091f9870e2 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badcd8 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394857:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25200/0x531a24091f987240 lrc: 0/0,0 mode: --/PR res: [0x240000400:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae03c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827454.394911:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000078f89158/0x531a24091f9870c6 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badc92 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394915:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24a00/0x531a24091f987239 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae02e expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.394938:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000b0ab7367/0x531a24091f987064 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x20:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badb96 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394952:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4a00/0x531a24091f987208 lrc: 0/0,0 mode: --/PR res: [0x240000401:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badfb0 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827454.394984:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000ec26f951/0x531a24091f986fed lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bada62 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.394994:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4c00/0x531a24091f9871ec lrc: 0/0,0 mode: --/PR res: [0x240000401:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf6a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827454.394997:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000001fcc4b57/0x531a24091f986fdf lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bada3f expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.395043:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000c1a95828/0x531a24091f986f99 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad990 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395046:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4400/0x531a24091f9871d7 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x1c:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf32 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.395077:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000dba61ca0/0x531a24091f986f68 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x2d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad919 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.395111:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000c5186ada/0x531a24091f986f5a lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad8f6 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395113:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5200/0x531a24091f9871c9 lrc: 0/0,0 mode: --/PR res: [0x240000400:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf16 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.395157:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000059be989b/0x531a24091f986f3e lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x16:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad8b0 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395166:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5600/0x531a24091f9871c2 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x1c:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf08 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.395202:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000003578f3f5/0x531a24091f986f30 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad88d expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395209:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2200/0x531a24091f98719f lrc: 0/0,0 mode: --/PR res: [0x240000401:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badead expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827454.395252:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000017bcc6f2/0x531a24091f986a4b lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x473:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bac53f expref: -99 pid: 71124 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827454.395259:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2a00/0x531a24091f98718a lrc: 0/0,0 mode: --/PR res: [0x240000402:0x14:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade75 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.395304:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000aadaddaf/0x531a24091f986ed5 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x472:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad798 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395307:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2400/0x531a24091f98717c lrc: 0/0,0 mode: --/PR res: [0x240000400:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade59 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.395352:0:15688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000d8c8a625/0x531a24091f986eb9 lrc: 1/0,0 mode: --/PR res: [0x2c0000401:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad752 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395376:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb3600/0x531a24091f987175 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x14:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade4b expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.395394:0:15688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 27 locks packed 00010000:00010000:1.0:1787827454.395404:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478800/0x531a24091f987087 lrc: 0/0,0 mode: --/PR res: [0x240000401:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badbf8 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1787827454.395414:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000eeff973a/0x531a24091f987351 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x96:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae2e3 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395438:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478a00/0x531a24091f987025 lrc: 0/0,0 mode: --/PR res: [0x240000401:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badafc expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827454.395490:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007e00/0x531a24091f987009 lrc: 0/0,0 mode: --/PR res: [0x240000404:0x6b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badab6 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.395491:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000bd5fa4e2/0x531a24091f987343 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x2c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae2c0 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.395575:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000e3e3fa4d/0x531a24091f987335 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x2b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae29d expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395577:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007200/0x531a24091f986fbc lrc: 0/0,0 mode: --/PR res: [0x240000401:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad9eb expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827454.395636:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf600/0x531a24091f986f8b lrc: 0/0,0 mode: --/PR res: [0x240000402:0x6:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad96d expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.395648:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000002719a2a1/0x531a24091f987327 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x10:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae27a expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395653:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbee00/0x531a24091f986f7d lrc: 0/0,0 mode: --/PR res: [0x240000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad951 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00080000:3.0:1787827454.395681:0:2414:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9c1a49d8f480 x1874668972790528/t0(0) o103->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:17/18 lens 496/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1787827454.395698:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000ecd06b33/0x531a24091f9872cc lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x36:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae19a expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395713:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf200/0x531a24091f986f76 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad943 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827454.395771:0:15103:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1800/0x531a24091f986e9d lrc: 0/0,0 mode: --/PR res: [0x240000401:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad71a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.395789:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000effe73fa/0x531a24091f9872be lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x2e:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae177 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827454.395844:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827454.395852:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827454.395856:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827454.395860:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000d351dd5a/0x531a24091f9872b0 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae154 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00080000:1.0:1787827454.395880:0:2411:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9c1a78a78b40 x1874668972790272/t0(0) o103->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:17/18 lens 528/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827454.395949:0:2411:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1787827454.395968:0:2411:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9c1a788603c0 x1874668972790400/t0(0) o103->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:17/18 lens 528/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1787827454.395970:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000da0cc5fe/0x531a24091f98726a lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae09e expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396045:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000aaf421f1/0x531a24091f98725c lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x38:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bae07b expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396070:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000005f53f123/0x531a24091f987167 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x13:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bade21 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396104:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000009a1cc7f6/0x531a24091f987159 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x27:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753baddfe expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396153:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000014246f40/0x531a24091f98714b lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badddb expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396221:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000087ec420c/0x531a24091f987105 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badd25 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396276:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000b667ce01/0x531a24091f9870f7 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x2a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badd02 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396343:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000bf78db19/0x531a24091f9870e2 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x12:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badcd8 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396406:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000078f89158/0x531a24091f9870c6 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badc92 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396420:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000b0ab7367/0x531a24091f987064 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x20:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753badb96 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396442:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000ec26f951/0x531a24091f986fed lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x26:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bada62 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396473:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000001fcc4b57/0x531a24091f986fdf lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x28:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bada3f expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396532:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000c1a95828/0x531a24091f986f99 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x29:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad990 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396544:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000dba61ca0/0x531a24091f986f68 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x2d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad919 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396578:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000c5186ada/0x531a24091f986f5a lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x39:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad8f6 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396631:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000059be989b/0x531a24091f986f3e lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x16:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad8b0 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396703:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 000000003578f3f5/0x531a24091f986f30 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x35:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad88d expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396748:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 0000000017bcc6f2/0x531a24091f986a4b lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x473:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bac53f expref: -99 pid: 71124 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1787827454.396769:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000aadaddaf/0x531a24091f986ed5 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x472:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad798 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396831:0:15688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff9c1a49bbf800 lock: 00000000d8c8a625/0x531a24091f986eb9 lrc: 0/0,0 mode: --/PR res: [0x2c0000401:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0x9505c05753bad752 expref: -99 pid: 80918 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1787827454.396847:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827454.396851:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1787827454.396857:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827455.159491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827455.415470:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827456.183441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827456.439506:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827457.207452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827457.463515:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827457.719463:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827457.719471:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827457.719475:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827457.719484:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3908 00000100:00080000:1.0:1787827457.719494:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827457.719502:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827457.719556:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972790656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827457.719582:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827457.719584:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827457.719586:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827457.719589:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3908 00000100:00080000:1.0:1787827457.719593:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827457.719597:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827457.719605:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827457.719610:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972790656:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827457.719619:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972790784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827457.719628:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827457.719630:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827457.719631:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827457.719633:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3908 00000100:00080000:1.0:1787827457.719635:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827457.719638:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827457.719641:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827457.719648:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972790912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827457.719659:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827457.719662:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1787827457.719665:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827457.719666:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827457.719672:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3908 00000100:00080000:1.0:1787827457.719675:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827457.719678:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827457.719681:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827457.719688:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972791040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827457.719694:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827457.719697:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827457.719699:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827457.719703:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827457/real 1787827457] req@ffff9c1a78a79a40 x1874668972790656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827457.719704:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3908 00000100:00080000:1.0:1787827457.719710:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827457.719715:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827457.719721:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827457.719727:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972790656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827457.719731:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972791168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827457.719756:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827457.719762:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827457.719764:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827457.719764:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827457.719773:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827457.719774:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3908 00000100:00080000:1.0:1787827457.719779:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827457.719781:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972790656:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827457.719785:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827457.719790:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827457.719797:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972791296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827457.719805:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827457.719805:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972790784:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827457.719812:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827457.719814:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827457.719817:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3908 00000100:00080000:1.0:1787827457.719820:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827457.719822:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827457.719826:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827457.719830:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827457.719836:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972791424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827457.719843:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827457/real 1787827457] req@ffff9c1a78a7b840 x1874668972790784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827457.719855:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972790784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827457.719862:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827457.719865:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827457.719867:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972790784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827457.719872:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972790912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827457.719883:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827457.719891:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827457/real 1787827457] req@ffff9c1a78a792c0 x1874668972790912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827457.719903:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972790912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827457.719912:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827457.719916:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827457.719919:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972790912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827457.719927:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972791040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827457.719947:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827457.719959:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827457/real 1787827457] req@ffff9c1a78a7b480 x1874668972791040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827457.719969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972791040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827457.719974:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827457.719976:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827457.719979:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972791040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827457.719983:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972791168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827457.719996:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827457.720003:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827457/real 1787827457] req@ffff9c1a78a78000 x1874668972791168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827457.720010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972791168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827457.720015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827457.720016:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827457.720018:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972791168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827457.720030:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972791296:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827457.720041:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827457.720051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827457/real 1787827457] req@ffff9c1a78a7a580 x1874668972791296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827457.720064:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972791296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827457.720073:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827457.720076:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827457.720079:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972791296:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827457.720085:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972791424:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827457.720103:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827457.720114:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827457/real 1787827457] req@ffff9c1a78a78f00 x1874668972791424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827457.720126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972791424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827457.720132:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827457.720134:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827457.720136:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972791424:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827458.231442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827458.487547:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827459.255421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827459.511514:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827460.279435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827460.535535:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827461.303453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827461.559610:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827461.560248:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8d800/0x531a24091f9870b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753badc68 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560254:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x62:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560261:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0e00/0x531a24091f9870a3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x166:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753badc45 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560267:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x166:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560273:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478200/0x531a24091f987095 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753badc22 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560279:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x60:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560285:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479200/0x531a24091f98708e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x23:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badc06 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560295:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478c00/0x531a24091f987080 lrc: 2/0,0 mode: PR/PR res: [0x200000400:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badbea expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.560301:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479400/0x531a24091f987079 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x23:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badbdc expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560311:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478600/0x531a24091f987072 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x8a:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badbc0 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560320:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478000/0x531a24091f98706b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x8a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badbb2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560329:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0a00/0x531a24091f98705d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753badb8f expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560334:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x4f:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560341:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478e00/0x531a24091f98704f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753badb6c expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560371:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x30:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560380:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1a00/0x531a24091f987041 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753badb49 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560389:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5c:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560397:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479000/0x531a24091f987033 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753badb26 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560405:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x68:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560415:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479c00/0x531a24091f98702c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x25:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badb0a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560425:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479800/0x531a24091f98701e lrc: 2/0,0 mode: PR/PR res: [0x200000400:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badaee expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.560433:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479600/0x531a24091f987017 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x25:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badae0 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560442:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006000/0x531a24091f987010 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x164:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badac4 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560452:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006c00/0x531a24091f987002 lrc: 2/0,0 mode: PR/PR res: [0x200000bd0:0x6b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753badaa8 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560459:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006400/0x531a24091f986ffb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x164:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bada9a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560472:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007800/0x531a24091f986ff4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bada7e expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560478:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x7a:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560484:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007a00/0x531a24091f986fe6 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bada5b expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560490:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x59:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560496:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007400/0x531a24091f986fd8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bada38 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560518:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5d:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560527:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006800/0x531a24091f986fca lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x169:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bada15 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560534:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x169:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560540:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006200/0x531a24091f986fc3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x7e:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad9f9 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560550:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006600/0x531a24091f986fb5 lrc: 2/0,0 mode: PR/PR res: [0x200000400:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad9dd expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560559:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007600/0x531a24091f986fae lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x7e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad9cf expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560568:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf400/0x531a24091f986fa0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad9ac expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560576:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x3c:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560582:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbfc00/0x531a24091f986f92 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad989 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560588:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5f:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560595:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf800/0x531a24091f986f84 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad95f expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.560603:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe800/0x531a24091f986f6f lrc: 2/0,0 mode: PR/PR res: [0x240000402:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad935 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560608:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe000/0x531a24091f986f61 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x67:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad912 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560614:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x67:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560620:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe600/0x531a24091f986f53 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad8ef expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560626:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x85:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560633:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe200/0x531a24091f986f45 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad8cc expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560639:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x64:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560645:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf000/0x531a24091f986f37 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad8a9 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560650:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x3b:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560656:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9c00/0x531a24091f986f29 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad886 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560663:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x78:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560669:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8200/0x531a24091f986f22 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b78:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad86a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560678:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8a00/0x531a24091f986f1b lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x1b78:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad85c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560687:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8800/0x531a24091f986f14 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x16a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad840 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560693:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x16a:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560698:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8600/0x531a24091f986f06 lrc: 2/0,0 mode: PR/PR res: [0x200000403:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad81d expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560704:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000403:0x29:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560710:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9000/0x531a24091f986eff lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x48:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad801 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560720:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9600/0x531a24091f986ef1 lrc: 2/0,0 mode: PR/PR res: [0x200000400:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad7e5 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.560727:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8e00/0x531a24091f986eea lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x48:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad7d7 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560737:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8000/0x531a24091f986edc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad7b4 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560743:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x66:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560754:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9e00/0x531a24091f986ece lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x168:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad791 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560765:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x168:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560777:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0200/0x531a24091f986ec0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad76e expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560787:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x5:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560798:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1e00/0x531a24091f986eab lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x9505c05753bad744 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827461.560805:0:15689:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x13:0x0]: losing layout lock 00010000:00010000:2.0:1787827461.560811:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0800/0x531a24091f986ea4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x75:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad728 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560821:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1000/0x531a24091f986e96 lrc: 2/0,0 mode: PR/PR res: [0x200000400:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad70c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560829:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1200/0x531a24091f986e8f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x75:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad6fe expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560838:0:15689:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1400/0x531a24091f986e88 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x9505c05753bad6e2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560870:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fce00/0x531a24091f987358 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae2ff expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560875:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7c00/0x531a24091f98734a lrc: 1/0,0 mode: --/PR res: [0x200000402:0x163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae2dc expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560883:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fd200/0x531a24091f98733c lrc: 1/0,0 mode: --/PR res: [0x200000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae2b9 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560887:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8c600/0x531a24091f98732e lrc: 1/0,0 mode: --/PR res: [0x200000402:0x63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae296 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560891:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fdc00/0x531a24091f987320 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae273 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560895:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fc200/0x531a24091f987312 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae249 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560899:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fca00/0x531a24091f9872fd lrc: 1/0,0 mode: --/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae21f expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560903:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fc400/0x531a24091f9872f6 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x21:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae203 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560907:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fd800/0x531a24091f9872e8 lrc: 1/0,0 mode: --/PR res: [0x200000400:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae1e7 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.560912:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e20400/0x531a24091f9872e1 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x21:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae1d9 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560916:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4167b000/0x531a24091f9872d3 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae1b6 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560920:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4167a400/0x531a24091f9872c5 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae193 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560925:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4167a200/0x531a24091f9872b7 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae170 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560929:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4167a800/0x531a24091f9872a9 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae14d expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560933:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4167ba00/0x531a24091f9872a2 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae131 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560937:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24c00/0x531a24091f987294 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae10e expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560941:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25000/0x531a24091f987286 lrc: 1/0,0 mode: --/PR res: [0x200000401:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae0e4 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560945:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24800/0x531a24091f987271 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x28:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae0ba expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560949:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24600/0x531a24091f987263 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae097 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560953:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24200/0x531a24091f987255 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae074 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560957:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25400/0x531a24091f987247 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae04a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.560961:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25600/0x531a24091f987232 lrc: 1/0,0 mode: --/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae020 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560966:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4e00/0x531a24091f987224 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x7d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badffd expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560971:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4800/0x531a24091f987216 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badfda expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560975:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4600/0x531a24091f98720f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x4a:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badfbe expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560979:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5c00/0x531a24091f987201 lrc: 1/0,0 mode: --/PR res: [0x200000400:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badfa2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.560983:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5a00/0x531a24091f9871fa lrc: 1/0,0 mode: --/PR res: [0x200000402:0x4a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf94 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560987:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5400/0x531a24091f9871f3 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x24:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf78 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.560991:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4200/0x531a24091f9871e5 lrc: 1/0,0 mode: --/PR res: [0x200000400:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf5c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.560995:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5000/0x531a24091f9871de lrc: 1/0,0 mode: --/PR res: [0x200000402:0x24:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf4e expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.560999:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5800/0x531a24091f9871d0 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf24 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561003:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2e00/0x531a24091f9871bb lrc: 1/0,0 mode: --/PR res: [0x240000402:0x1c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badefa expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561008:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2000/0x531a24091f9871ad lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753baded7 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561012:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb3000/0x531a24091f9871a6 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x22:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badebb expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561016:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2600/0x531a24091f987198 lrc: 1/0,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade9f expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561020:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2c00/0x531a24091f987191 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x22:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade91 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561024:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb3e00/0x531a24091f987183 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade67 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561028:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb3400/0x531a24091f98716e lrc: 1/0,0 mode: --/PR res: [0x240000402:0x14:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade3d expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561032:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a40031400/0x531a24091f987160 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bade1a expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561036:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e20800/0x531a24091f987152 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753baddf7 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561040:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a40031800/0x531a24091f987144 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753baddd4 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561044:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8de00/0x531a24091f987136 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753baddb1 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561049:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8d200/0x531a24091f98711a lrc: 1/0,0 mode: --/PR res: [0x200000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badd6b expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561054:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8da00/0x531a24091f987113 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x28:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badd4f expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561058:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8cc00/0x531a24091f98710c lrc: 1/0,0 mode: --/PR res: [0x200000403:0x28:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badd41 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561062:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a40031600/0x531a24091f9870fe lrc: 1/0,0 mode: --/PR res: [0x200000403:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badd1e expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561066:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e20c00/0x531a24091f9870f0 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x61:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badcfb expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561070:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8d600/0x531a24091f9870db lrc: 1/0,0 mode: --/PR res: [0x200000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badcd1 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561074:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8c200/0x531a24091f9870cd lrc: 1/0,0 mode: --/PR res: [0x200000403:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badcae expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561078:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a40031e00/0x531a24091f9870bf lrc: 1/0,0 mode: --/PR res: [0x200000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badc8b expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561082:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8d800/0x531a24091f9870b1 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badc68 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561086:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0e00/0x531a24091f9870a3 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x166:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badc45 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561091:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478200/0x531a24091f987095 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badc22 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561095:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479200/0x531a24091f98708e lrc: 1/0,0 mode: --/PR res: [0x200000402:0x23:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badc06 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561099:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478c00/0x531a24091f987080 lrc: 1/0,0 mode: --/PR res: [0x200000400:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badbea expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561103:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479400/0x531a24091f987079 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x23:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badbdc expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561107:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478600/0x531a24091f987072 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x8a:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badbc0 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561111:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478000/0x531a24091f98706b lrc: 1/0,0 mode: --/PR res: [0x200000402:0x8a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badbb2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561115:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0a00/0x531a24091f98705d lrc: 1/0,0 mode: --/PR res: [0x200000402:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badb8f expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561119:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478e00/0x531a24091f98704f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badb6c expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561123:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1a00/0x531a24091f987041 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badb49 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561127:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479000/0x531a24091f987033 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x68:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badb26 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561131:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479c00/0x531a24091f98702c lrc: 1/0,0 mode: --/PR res: [0x200000402:0x25:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badb0a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561135:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479800/0x531a24091f98701e lrc: 1/0,0 mode: --/PR res: [0x200000400:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badaee expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561139:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479600/0x531a24091f987017 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x25:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badae0 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561143:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006000/0x531a24091f987010 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x164:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badac4 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561147:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006c00/0x531a24091f987002 lrc: 1/0,0 mode: --/PR res: [0x200000bd0:0x6b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badaa8 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561151:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006400/0x531a24091f986ffb lrc: 1/0,0 mode: --/PR res: [0x200000402:0x164:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bada9a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561155:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007800/0x531a24091f986ff4 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bada7e expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561159:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007a00/0x531a24091f986fe6 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bada5b expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561163:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007400/0x531a24091f986fd8 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bada38 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561167:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006800/0x531a24091f986fca lrc: 1/0,0 mode: --/PR res: [0x200000402:0x169:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bada15 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561172:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006200/0x531a24091f986fc3 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x7e:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad9f9 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561176:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006600/0x531a24091f986fb5 lrc: 1/0,0 mode: --/PR res: [0x200000400:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad9dd expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561180:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007600/0x531a24091f986fae lrc: 1/0,0 mode: --/PR res: [0x200000402:0x7e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad9cf expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561184:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf400/0x531a24091f986fa0 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad9ac expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561188:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbfc00/0x531a24091f986f92 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad989 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561192:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf800/0x531a24091f986f84 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad95f expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561196:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe800/0x531a24091f986f6f lrc: 1/0,0 mode: --/PR res: [0x240000402:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad935 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561200:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe000/0x531a24091f986f61 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x67:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad912 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561204:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe600/0x531a24091f986f53 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad8ef expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561208:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe200/0x531a24091f986f45 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad8cc expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561212:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf000/0x531a24091f986f37 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad8a9 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561216:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9c00/0x531a24091f986f29 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad886 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561220:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8200/0x531a24091f986f22 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1b78:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad86a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561224:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8a00/0x531a24091f986f1b lrc: 1/0,0 mode: --/PR res: [0x200000403:0x1b78:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad85c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561228:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8800/0x531a24091f986f14 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x16a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad840 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561232:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8600/0x531a24091f986f06 lrc: 1/0,0 mode: --/PR res: [0x200000403:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad81d expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561236:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9000/0x531a24091f986eff lrc: 1/0,0 mode: --/PR res: [0x200000402:0x48:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad801 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561240:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9600/0x531a24091f986ef1 lrc: 1/0,0 mode: --/PR res: [0x200000400:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad7e5 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561244:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8e00/0x531a24091f986eea lrc: 1/0,0 mode: --/PR res: [0x200000402:0x48:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad7d7 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561248:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8000/0x531a24091f986edc lrc: 1/0,0 mode: --/PR res: [0x200000402:0x66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad7b4 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561252:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9e00/0x531a24091f986ece lrc: 1/0,0 mode: --/PR res: [0x200000402:0x168:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad791 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561256:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0200/0x531a24091f986ec0 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad76e expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561260:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1e00/0x531a24091f986eab lrc: 1/0,0 mode: --/PR res: [0x200000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad744 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561264:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0800/0x531a24091f986ea4 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x75:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad728 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561268:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1000/0x531a24091f986e96 lrc: 1/0,0 mode: --/PR res: [0x200000400:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad70c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561272:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1200/0x531a24091f986e8f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x75:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad6fe expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561276:0:15689:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1400/0x531a24091f986e88 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad6e2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561280:0:15689:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 99 locks packed 00010000:00010000:2.0:1787827461.561301:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fce00/0x531a24091f987358 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae2ff expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561313:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46ef7c00/0x531a24091f98734a lrc: 0/0,0 mode: --/PR res: [0x200000402:0x163:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae2dc expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561324:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fd200/0x531a24091f98733c lrc: 0/0,0 mode: --/PR res: [0x200000402:0x65:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae2b9 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561331:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8c600/0x531a24091f98732e lrc: 0/0,0 mode: --/PR res: [0x200000402:0x63:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae296 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561339:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fdc00/0x531a24091f987320 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae273 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561365:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fc200/0x531a24091f987312 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae249 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561373:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fca00/0x531a24091f9872fd lrc: 0/0,0 mode: --/PR res: [0x240000402:0x13:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae21f expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00080000:3.0:1787827461.561377:0:2413:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9c1a49d8e940 x1874668972791552/t0(0) o103->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:17/18 lens 1104/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1787827461.561385:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fc400/0x531a24091f9872f6 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x21:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae203 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561393:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4a1fd800/0x531a24091f9872e8 lrc: 0/0,0 mode: --/PR res: [0x200000400:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae1e7 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561400:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e20400/0x531a24091f9872e1 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x21:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae1d9 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561408:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4167b000/0x531a24091f9872d3 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae1b6 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561420:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4167a400/0x531a24091f9872c5 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae193 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561427:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4167a200/0x531a24091f9872b7 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae170 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561433:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4167a800/0x531a24091f9872a9 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae14d expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561439:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a4167ba00/0x531a24091f9872a2 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae131 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561446:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24c00/0x531a24091f987294 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae10e expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561452:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25000/0x531a24091f987286 lrc: 0/0,0 mode: --/PR res: [0x200000401:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae0e4 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561459:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24800/0x531a24091f987271 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x28:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae0ba expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561464:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24600/0x531a24091f987263 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae097 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561470:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b24200/0x531a24091f987255 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bae074 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561480:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25400/0x531a24091f987247 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x4:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae04a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561486:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43b25600/0x531a24091f987232 lrc: 0/0,0 mode: --/PR res: [0x240000402:0x4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bae020 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561492:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4e00/0x531a24091f987224 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x7d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badffd expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561498:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4800/0x531a24091f987216 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badfda expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561516:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4600/0x531a24091f98720f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x4a:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badfbe expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561521:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5c00/0x531a24091f987201 lrc: 0/0,0 mode: --/PR res: [0x200000400:0xa:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badfa2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561527:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5a00/0x531a24091f9871fa lrc: 0/0,0 mode: --/PR res: [0x200000402:0x4a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf94 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561533:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5400/0x531a24091f9871f3 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x24:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf78 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561539:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa4200/0x531a24091f9871e5 lrc: 0/0,0 mode: --/PR res: [0x200000400:0x7:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf5c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561549:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5000/0x531a24091f9871de lrc: 0/0,0 mode: --/PR res: [0x200000402:0x24:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf4e expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561555:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a46fa5800/0x531a24091f9871d0 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badf24 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561560:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2e00/0x531a24091f9871bb lrc: 0/0,0 mode: --/PR res: [0x240000402:0x1c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badefa expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561566:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2000/0x531a24091f9871ad lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753baded7 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561573:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb3000/0x531a24091f9871a6 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x22:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badebb expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561578:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2600/0x531a24091f987198 lrc: 0/0,0 mode: --/PR res: [0x200000400:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade9f expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561584:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb2c00/0x531a24091f987191 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x22:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade91 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561589:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb3e00/0x531a24091f987183 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade67 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561596:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a49fb3400/0x531a24091f98716e lrc: 0/0,0 mode: --/PR res: [0x240000402:0x14:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bade3d expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561606:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a40031400/0x531a24091f987160 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bade1a expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561613:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e20800/0x531a24091f987152 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753baddf7 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561619:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a40031800/0x531a24091f987144 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753baddd4 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561625:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8de00/0x531a24091f987136 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753baddb1 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561631:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8d200/0x531a24091f98711a lrc: 0/0,0 mode: --/PR res: [0x200000402:0x69:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badd6b expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561637:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8da00/0x531a24091f987113 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x28:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badd4f expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561642:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8cc00/0x531a24091f98710c lrc: 0/0,0 mode: --/PR res: [0x200000403:0x28:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badd41 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561648:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a40031600/0x531a24091f9870fe lrc: 0/0,0 mode: --/PR res: [0x200000403:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badd1e expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561654:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a43e20c00/0x531a24091f9870f0 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x61:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badcfb expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561663:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8d600/0x531a24091f9870db lrc: 0/0,0 mode: --/PR res: [0x200000402:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badcd1 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561670:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8c200/0x531a24091f9870cd lrc: 0/0,0 mode: --/PR res: [0x200000403:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badcae expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561676:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a40031e00/0x531a24091f9870bf lrc: 0/0,0 mode: --/PR res: [0x200000402:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badc8b expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561682:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7ad8d800/0x531a24091f9870b1 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x62:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badc68 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561688:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0e00/0x531a24091f9870a3 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x166:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badc45 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561694:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478200/0x531a24091f987095 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x60:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badc22 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561700:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479200/0x531a24091f98708e lrc: 0/0,0 mode: --/PR res: [0x200000402:0x23:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badc06 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561705:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478c00/0x531a24091f987080 lrc: 0/0,0 mode: --/PR res: [0x200000400:0x6:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badbea expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561711:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479400/0x531a24091f987079 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x23:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badbdc expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561721:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478600/0x531a24091f987072 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x8a:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badbc0 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561726:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478000/0x531a24091f98706b lrc: 0/0,0 mode: --/PR res: [0x200000402:0x8a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badbb2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561731:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0a00/0x531a24091f98705d lrc: 0/0,0 mode: --/PR res: [0x200000402:0x4f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badb8f expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561737:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f478e00/0x531a24091f98704f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badb6c expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561744:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1a00/0x531a24091f987041 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badb49 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561755:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479000/0x531a24091f987033 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x68:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753badb26 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561765:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479c00/0x531a24091f98702c lrc: 0/0,0 mode: --/PR res: [0x200000402:0x25:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badb0a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561774:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479800/0x531a24091f98701e lrc: 0/0,0 mode: --/PR res: [0x200000400:0x8:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badaee expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561784:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7f479600/0x531a24091f987017 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x25:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badae0 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561799:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006000/0x531a24091f987010 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x164:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badac4 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561805:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006c00/0x531a24091f987002 lrc: 0/0,0 mode: --/PR res: [0x200000bd0:0x6b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753badaa8 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561811:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006400/0x531a24091f986ffb lrc: 0/0,0 mode: --/PR res: [0x200000402:0x164:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bada9a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561817:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007800/0x531a24091f986ff4 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bada7e expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561823:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007a00/0x531a24091f986fe6 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x59:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bada5b expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561829:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007400/0x531a24091f986fd8 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bada38 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561835:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006800/0x531a24091f986fca lrc: 0/0,0 mode: --/PR res: [0x200000402:0x169:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bada15 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561841:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006200/0x531a24091f986fc3 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x7e:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad9f9 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561846:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d006600/0x531a24091f986fb5 lrc: 0/0,0 mode: --/PR res: [0x200000400:0xc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad9dd expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561855:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7d007600/0x531a24091f986fae lrc: 0/0,0 mode: --/PR res: [0x200000402:0x7e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad9cf expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561861:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf400/0x531a24091f986fa0 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad9ac expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561868:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbfc00/0x531a24091f986f92 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad989 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561873:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf800/0x531a24091f986f84 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x5:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad95f expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561879:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe800/0x531a24091f986f6f lrc: 0/0,0 mode: --/PR res: [0x240000402:0x6:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad935 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561886:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe000/0x531a24091f986f61 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x67:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad912 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561892:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe600/0x531a24091f986f53 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad8ef expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561897:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbe200/0x531a24091f986f45 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x64:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad8cc expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561903:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7adbf000/0x531a24091f986f37 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad8a9 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561913:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9c00/0x531a24091f986f29 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad886 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561919:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8200/0x531a24091f986f22 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1b78:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad86a expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561924:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8a00/0x531a24091f986f1b lrc: 0/0,0 mode: --/PR res: [0x200000403:0x1b78:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad85c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561930:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8800/0x531a24091f986f14 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x16a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad840 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561936:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8600/0x531a24091f986f06 lrc: 0/0,0 mode: --/PR res: [0x200000403:0x29:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad81d expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561942:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9000/0x531a24091f986eff lrc: 0/0,0 mode: --/PR res: [0x200000402:0x48:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad801 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561946:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9600/0x531a24091f986ef1 lrc: 0/0,0 mode: --/PR res: [0x200000400:0x9:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad7e5 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827461.561952:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8e00/0x531a24091f986eea lrc: 0/0,0 mode: --/PR res: [0x200000402:0x48:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad7d7 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561958:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba8000/0x531a24091f986edc lrc: 0/0,0 mode: --/PR res: [0x200000402:0x66:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad7b4 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561967:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a7aba9e00/0x531a24091f986ece lrc: 0/0,0 mode: --/PR res: [0x200000402:0x168:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad791 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561973:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0200/0x531a24091f986ec0 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad76e expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561980:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1e00/0x531a24091f986eab lrc: 0/0,0 mode: --/PR res: [0x200000402:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x9505c05753bad744 expref: -99 pid: 80918 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561986:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d0800/0x531a24091f986ea4 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x75:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad728 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.561990:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1000/0x531a24091f986e96 lrc: 0/0,0 mode: --/PR res: [0x200000400:0xb:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad70c expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.561996:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1200/0x531a24091f986e8f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x75:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad6fe expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827461.562002:0:15689:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9c1a49bbf800 lock: ffff9c1a496d1400/0x531a24091f986e88 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x9505c05753bad6e2 expref: -99 pid: 80918 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827461.562008:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827461.562009:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827461.562010:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827462.327434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827462.327438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827462.583491:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827462.839469:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827462.839480:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827462.839484:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827462.839506:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3913 00000100:00080000:2.0:1787827462.839516:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827462.839524:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827462.839557:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972791680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827462.839578:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827462.839581:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827462.839582:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827462.839586:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3913 00000100:00080000:2.0:1787827462.839588:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827462.839591:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827462.839598:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827462.839607:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972791808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827462.839615:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827462.839617:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827462.839619:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827462.839623:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3913 00000100:00100000:3.0:1787827462.839625:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972791680:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827462.839626:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827462.839632:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827462.839636:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827462.839644:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972791936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827462.839652:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827462.839655:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827462.839657:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827462.839659:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3913 00000100:00080000:2.0:1787827462.839677:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827462.839680:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827462.839683:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827462.839689:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827462.839692:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972792064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827462.839700:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827462.839702:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827462.839704:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827462.839707:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3913 00000100:00080000:2.0:1787827462.839710:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827462.839712:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827462.839716:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827462.839723:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972792192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827462.839729:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827462.839733:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787827462.839735:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827462/real 1787827462] req@ffff9c1a49d8cb40 x1874668972791680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827462.839736:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827462.839742:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3913 00000100:00080000:2.0:1787827462.839755:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827462.839760:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827462.839761:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972791680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827462.839764:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827462.839775:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972792320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827462.839779:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827462.839783:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827462.839787:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827462.839789:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827462.839790:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827462.839797:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3913 00000100:00080000:2.0:1787827462.839801:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827462.839801:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972791680:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827462.839807:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827462.839812:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827462.839820:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972792448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827462.839835:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972791808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827462.839863:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827462.839877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827462/real 1787827462] req@ffff9c1a49d8ed00 x1874668972791808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827462.839890:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972791808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827462.839898:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827462.839902:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827462.839905:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972791808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827462.839911:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972791936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827462.839931:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827462.839942:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827462/real 1787827462] req@ffff9c1a49d8c3c0 x1874668972791936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827462.839954:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972791936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827462.839962:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827462.839965:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827462.839968:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972791936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827462.839975:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972792064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827462.839992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827462.840001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827462/real 1787827462] req@ffff9c1a49d8e580 x1874668972792064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827462.840013:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972792064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827462.840021:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827462.840024:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827462.840027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972792064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827462.840033:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972792192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827462.840051:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827462.840062:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827462/real 1787827462] req@ffff9c1a49d8fc00 x1874668972792192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827462.840074:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972792192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827462.840088:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827462.840091:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827462.840095:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972792192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827462.840101:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972792320:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827462.840121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827462.840132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827462/real 1787827462] req@ffff9c1a49d8e1c0 x1874668972792320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827462.840144:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972792320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827462.840152:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827462.840155:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827462.840158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972792320:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827462.840162:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972792448:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827462.840176:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827462.840183:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827462/real 1787827462] req@ffff9c1a49d8c000 x1874668972792448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827462.840190:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972792448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827462.840195:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827462.840197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827462.840200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972792448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827463.351429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827463.351429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827463.607516:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827464.375440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827464.375440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827464.631464:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827465.399484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827465.399484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827465.655597:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827466.423416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827466.423452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827466.679539:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827467.447478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827467.447478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827467.703604:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827467.959517:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827467.959529:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827467.959535:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827467.959548:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3918 00000100:00080000:2.0:1787827467.959563:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827467.959572:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827467.959621:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972792576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827467.959655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827467.959660:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827467.959662:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827467.959667:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3918 00000100:00080000:2.0:1787827467.959671:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827467.959675:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827467.959688:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827467.959703:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972792704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827467.959716:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972792576:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827467.959717:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827467.959727:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827467.959731:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827467.959737:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3918 00000100:00080000:2.0:1787827467.959741:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827467.959746:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827467.959752:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827467.959766:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972792832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827467.959779:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827467.959783:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827467.959784:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827467.959789:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3918 00000100:00080000:2.0:1787827467.959794:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827467.959799:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827467.959804:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827467.959808:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827467.959820:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972792960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827467.959831:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827467.959835:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827467.959837:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827467.959841:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3918 00000100:00080000:2.0:1787827467.959844:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827467.959848:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827467.959854:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827467.959863:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827467/real 1787827467] req@ffff9c1a49d8de00 x1874668972792576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827467.959868:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972793088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827467.959895:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827467.959898:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827467.959901:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827467.959903:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972792576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827467.959908:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3918 00000100:00080000:2.0:1787827467.959915:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827467.959923:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787827467.959926:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787827467.959929:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827467.959935:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827467.959943:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972792576:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827467.959947:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972793216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827467.959958:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827467.959962:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827467.959964:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827467.959968:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3918 00000100:00080000:2.0:1787827467.959972:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827467.959975:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827467.959980:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827467.959986:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972792704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827467.959995:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972793344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827467.960015:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827467.960027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827467/real 1787827467] req@ffff9c1a49d8da40 x1874668972792704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827467.960037:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972792704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827467.960044:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827467.960048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827467.960050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972792704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827467.960054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972792832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827467.960066:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827467.960073:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827467/real 1787827467] req@ffff9c1a49d8cf00 x1874668972792832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827467.960080:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972792832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827467.960085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827467.960087:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827467.960090:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972792832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827467.960096:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972792960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827467.960116:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827467.960128:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827467/real 1787827467] req@ffff9c1a49d8f0c0 x1874668972792960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827467.960141:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972792960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827467.960151:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827467.960154:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827467.960158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972792960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827467.960165:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972793088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827467.960186:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827467.960199:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827467/real 1787827467] req@ffff9c1a49d8d680 x1874668972793088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827467.960208:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972793088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827467.960214:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827467.960217:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827467.960219:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972793088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827467.960223:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972793216:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827467.960239:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827467.960245:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827467/real 1787827467] req@ffff9c1a49d8c780 x1874668972793216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827467.960253:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972793216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827467.960258:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827467.960260:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827467.960262:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972793216:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827467.960276:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972793344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827467.960287:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827467.960293:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827467/real 1787827467] req@ffff9c1a49d8de00 x1874668972793344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827467.960305:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972793344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827467.960311:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827467.960313:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827467.960314:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972793344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827468.471441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827468.471442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827468.727524:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827469.495519:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827469.495519:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827469.751561:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827470.519432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827470.520411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827470.775525:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827471.543493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827471.543499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827471.799605:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827472.567481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827472.567481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827472.823566:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827473.079465:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827473.079477:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827473.079483:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827473.079496:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3923 00000100:00080000:0.0:1787827473.079510:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827473.079520:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827473.079569:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972793472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827473.079604:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827473.079609:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827473.079612:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827473.079618:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3923 00000100:00080000:0.0:1787827473.079622:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827473.079626:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827473.079636:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827473.079651:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972793600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827473.079664:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827473.079666:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972793472:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827473.079668:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827473.079670:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827473.079681:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3923 00000100:00080000:0.0:1787827473.079685:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827473.079687:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827473.079691:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827473.079699:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972793728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827473.079712:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827473.079715:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827473.079718:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827473.079721:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3923 00000100:00080000:0.0:1787827473.079726:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827473.079729:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827473.079732:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827473.079743:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972793856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827473.079754:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827473.079761:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827473.079765:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827473.079767:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827473.079772:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3923 00000100:00080000:0.0:1787827473.079776:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827473.079779:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827473.079783:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827473.079793:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972793984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827473.079799:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827473.079802:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827473.079804:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827473.079808:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3923 00000100:00080000:0.0:1787827473.079810:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827473.079816:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827473.079819:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827473.079823:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827473/real 1787827473] req@ffff9c1a788621c0 x1874668972793472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827473.079835:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972794112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827473.079851:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827473.079853:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972793472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827473.079861:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827473.079869:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827473.079878:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3923 00000100:00080000:0.0:1787827473.079887:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827473.079892:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827473.079893:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827473.079909:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827473.079909:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827473.079928:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972793472:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827473.079936:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972794240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827473.079963:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972793600:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827473.079993:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827473.080006:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827473/real 1787827473] req@ffff9c1a78862580 x1874668972793600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827473.080023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972793600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827473.080032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827473.080036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827473.080039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972793600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827473.080047:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972793728:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827473.080077:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827473.080088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827473/real 1787827473] req@ffff9c1a78861680 x1874668972793728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827473.080107:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972793728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827473.080116:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827473.080120:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827473.080123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972793728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827473.080130:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972793856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827473.080147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827473.080157:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827473/real 1787827473] req@ffff9c1a788630c0 x1874668972793856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827473.080184:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972793856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827473.080194:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827473.080197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827473.080200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972793856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827473.080207:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972793984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827473.080224:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827473.080234:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827473/real 1787827473] req@ffff9c1a78863c00 x1874668972793984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827473.080245:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972793984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827473.080255:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827473.080258:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827473.080261:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972793984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827473.080267:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972794112:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827473.080283:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827473.080292:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827473/real 1787827473] req@ffff9c1a78860000 x1874668972794112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827473.080305:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972794112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827473.080320:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827473.080323:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827473.080326:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972794112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827473.080332:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972794240:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827473.080377:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827473.080388:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827473/real 1787827473] req@ffff9c1a78860780 x1874668972794240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827473.080400:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972794240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827473.080409:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827473.080412:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827473.080416:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972794240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827473.591480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827473.591494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827473.847604:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827474.615504:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827474.615522:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827474.871545:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827475.639489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827475.639509:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827475.895563:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827476.663462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827476.663483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827476.919561:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827477.687477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827477.687501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827477.943545:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827478.199471:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827478.199479:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827478.199483:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827478.199492:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3928 00000100:00080000:2.0:1787827478.199501:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827478.199506:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827478.199538:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972794368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827478.199559:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827478.199562:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827478.199563:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827478.199566:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3928 00000100:00080000:2.0:1787827478.199568:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827478.199570:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827478.199578:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827478.199587:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972794496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827478.199594:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827478.199597:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827478.199598:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827478.199600:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3928 00000100:00080000:2.0:1787827478.199602:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827478.199607:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827478.199608:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972794368:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787827478.199610:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827478.199621:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972794624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827478.199628:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827478.199631:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827478.199632:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827478.199634:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3928 00000100:00080000:2.0:1787827478.199637:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827478.199640:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827478.199643:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827478.199651:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972794752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827478.199657:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827478.199659:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827478.199661:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827478.199663:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3928 00000100:00080000:2.0:1787827478.199665:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827478.199669:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827478.199671:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827478.199676:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827478.199685:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972794880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827478.199691:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827478.199693:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827478.199694:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827478.199697:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3928 00000100:00080000:2.0:1787827478.199700:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827478.199703:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827478.199707:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827478.199709:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827478/real 1787827478] req@ffff9c1a49d8c000 x1874668972794368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827478.199718:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972795008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827478.199753:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827478.199756:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827478.199757:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827478.199761:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972794368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827478.199763:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3928 00000100:00080000:2.0:1787827478.199768:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827478.199775:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787827478.199778:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787827478.199779:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827478.199786:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827478.199791:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972795136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827478.199792:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972794368:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827478.199804:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972794496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827478.199826:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827478.199836:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827478/real 1787827478] req@ffff9c1a49d8e1c0 x1874668972794496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827478.199846:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972794496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827478.199851:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827478.199853:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827478.199855:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972794496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827478.199875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972794624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827478.199885:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827478.199891:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827478/real 1787827478] req@ffff9c1a49d8fc00 x1874668972794624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827478.199903:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972794624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827478.199918:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827478.199921:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827478.199925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972794624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827478.199943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972794752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827478.199963:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827478.199971:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827478/real 1787827478] req@ffff9c1a49d8e580 x1874668972794752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827478.199979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972794752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827478.199985:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827478.199987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827478.199989:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972794752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827478.199993:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972794880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827478.200003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827478.200009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827478/real 1787827478] req@ffff9c1a49d8c3c0 x1874668972794880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827478.200016:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972794880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827478.200021:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827478.200023:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827478.200025:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972794880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827478.200029:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972795008:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827478.200048:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827478.200054:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827478/real 1787827478] req@ffff9c1a49d8ed00 x1874668972795008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827478.200062:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972795008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827478.200067:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827478.200069:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827478.200071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972795008:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827478.200074:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972795136:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827478.200084:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827478.200089:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827478/real 1787827478] req@ffff9c1a49d8cb40 x1874668972795136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827478.200096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972795136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827478.200101:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827478.200103:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827478.200105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972795136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827478.711477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827478.711477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827478.967554:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827479.735471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827479.735471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827479.991560:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827480.759470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827480.759479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827481.015526:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827481.783557:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827481.783617:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827482.039595:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827482.807476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827482.807491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827483.063482:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827483.319525:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827483.319536:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827483.319541:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827483.319553:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3933 00000100:00080000:2.0:1787827483.319565:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827483.319572:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827483.319631:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972795264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827483.319672:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827483.319675:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827483.319677:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827483.319680:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3933 00000100:00080000:2.0:1787827483.319683:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827483.319687:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827483.319697:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827483.319708:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972795392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827483.319716:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827483.319719:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827483.319720:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827483.319725:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3933 00000100:00100000:3.0:1787827483.319727:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972795264:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827483.319728:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827483.319735:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827483.319739:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827483.319747:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972795520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827483.319755:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827483.319758:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827483.319759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827483.319762:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3933 00000100:00080000:2.0:1787827483.319764:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827483.319767:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827483.319770:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827483.319778:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972795648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827483.319790:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827483.319793:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827483.319794:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827483.319797:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3933 00000100:00080000:2.0:1787827483.319799:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827483.319801:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827483.319804:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827483.319809:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827483.319815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972795776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827483.319823:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827483.319826:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827483.319828:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827483.319831:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3933 00000100:00080000:2.0:1787827483.319834:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827483.319837:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827483.319841:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827483.319849:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972795904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827483.319854:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827483.319856:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827483.319857:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827483.319859:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3933 00000100:00080000:2.0:1787827483.319862:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827483.319866:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827483.319870:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827483.319878:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827483/real 1787827483] req@ffff9c1a49d8cb40 x1874668972795264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827483.319882:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972796032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827483.319908:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972795264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827483.319918:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827483.319922:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827483.319926:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972795264:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827483.319948:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972795392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827483.319972:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827483.319982:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827483/real 1787827483] req@ffff9c1a49d8ed00 x1874668972795392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827483.319991:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972795392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827483.319997:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827483.320001:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827483.320004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972795392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827483.320011:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972795520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827483.320032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827483.320044:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827483/real 1787827483] req@ffff9c1a49d8c3c0 x1874668972795520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827483.320058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972795520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827483.320064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827483.320066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827483.320068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972795520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827483.320073:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972795648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827483.320088:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827483.320096:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827483/real 1787827483] req@ffff9c1a49d8e580 x1874668972795648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827483.320103:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972795648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827483.320109:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827483.320111:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827483.320113:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972795648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827483.320116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972795776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827483.320126:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827483.320132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827483/real 1787827483] req@ffff9c1a49d8fc00 x1874668972795776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827483.320144:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972795776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827483.320150:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827483.320151:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827483.320153:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972795776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827483.320157:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972795904:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827483.320168:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827483.320174:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827483/real 1787827483] req@ffff9c1a49d8e1c0 x1874668972795904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827483.320181:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972795904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827483.320186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827483.320188:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827483.320190:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972795904:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827483.320194:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972796032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827483.320211:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827483.320222:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827483/real 1787827483] req@ffff9c1a49d8c000 x1874668972796032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827483.320236:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972796032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827483.320247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827483.320249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827483.320251:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972796032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827483.831436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827483.831486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827484.087545:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827484.855467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827484.855481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827485.111604:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827485.879419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827485.879466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827486.135550:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827486.903440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827486.903447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827487.159571:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827487.927402:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827487.927422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827488.183525:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827488.439461:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827488.439471:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827488.439475:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827488.439486:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3938 00000100:00080000:2.0:1787827488.439496:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827488.439504:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827488.439543:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972796160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827488.439613:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827488.439621:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827488.439626:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827488.439636:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3938 00000100:00080000:2.0:1787827488.439644:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827488.439654:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827488.439671:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827488.439700:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972796160:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827488.439719:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972796288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827488.439738:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827488.439743:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827488.439746:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827488.439753:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3938 00000100:00080000:2.0:1787827488.439762:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827488.439767:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827488.439774:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827488.439792:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972796416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827488.439798:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827488.439815:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827488.439819:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827488.439822:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827488.439828:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3938 00000100:00080000:2.0:1787827488.439832:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827488.439837:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827488.439843:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827488.439853:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827488/real 1787827488] req@ffff9c1a49d8de00 x1874668972796160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827488.439860:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972796544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827488.439900:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827488.439905:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827488.439909:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827488.439911:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972796160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827488.439918:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3938 00000100:00080000:2.0:1787827488.439927:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827488.439937:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787827488.439941:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787827488.439943:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787827488.439952:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827488.439961:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972796672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827488.439962:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972796160:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827488.440000:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827488.440003:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827488.440005:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827488.440009:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3938 00000100:00080000:2.0:1787827488.440013:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827488.440020:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972796288:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827488.440020:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827488.440031:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827488.440045:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972796800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827488.440076:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827488.440081:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827488.440085:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827488.440092:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827488.440096:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3938 00000100:00080000:2.0:1787827488.440104:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827488.440109:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827488.440116:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827488.440137:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827488/real 1787827488] req@ffff9c1a49d8c780 x1874668972796288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827488.440137:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972796928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827488.440189:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972796288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827488.440203:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827488.440208:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827488.440213:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972796288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827488.440222:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972796416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827488.440253:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827488.440269:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827488/real 1787827488] req@ffff9c1a49d8d680 x1874668972796416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827488.440285:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972796416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827488.440296:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827488.440301:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827488.440305:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972796416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827488.440336:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972796544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827488.440421:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827488.440441:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827488/real 1787827488] req@ffff9c1a49d8f0c0 x1874668972796544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827488.440459:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972796544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827488.440468:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827488.440471:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827488.440476:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972796544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827488.440485:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972796672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827488.440511:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827488.440527:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827488/real 1787827488] req@ffff9c1a49d8cf00 x1874668972796672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827488.440545:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972796672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827488.440574:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827488.440578:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827488.440583:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972796672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827488.440616:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972796800:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827488.440653:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827488.440674:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827488/real 1787827488] req@ffff9c1a49d8da40 x1874668972796800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827488.440714:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972796800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827488.440739:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827488.440743:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827488.440747:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972796800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827488.440754:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972796928:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827488.440779:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827488.440792:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827488/real 1787827488] req@ffff9c1a49d8de00 x1874668972796928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827488.440804:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972796928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827488.440813:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827488.440816:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827488.440820:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972796928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827488.951438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827488.951444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827489.207548:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827489.975443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827489.975443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827490.231491:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827490.999445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827490.999445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827491.255528:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827492.023459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827492.023460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827492.279470:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827493.047459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827493.047529:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827493.303586:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827493.559458:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827493.559471:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827493.559478:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827493.559497:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3943 00000100:00080000:2.0:1787827493.559513:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827493.559537:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827493.559594:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972797056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827493.559670:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827493.559677:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827493.559680:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827493.559688:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3943 00000100:00080000:2.0:1787827493.559696:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827493.559704:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827493.559756:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827493.559786:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972797184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827493.559805:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827493.559812:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827493.559816:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827493.559827:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3943 00000100:00100000:0.0:1787827493.559832:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972797056:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827493.559838:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827493.559859:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827493.559868:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827493.559893:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972797312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827493.559910:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827493.559918:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827493.559921:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827493.559929:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3943 00000100:00080000:2.0:1787827493.559939:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827493.559948:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827493.559958:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827493.559999:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827493.560002:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972797440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827493.560062:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827493.560066:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827493.560069:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827493.560077:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3943 00000100:00080000:2.0:1787827493.560084:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827493.560093:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827493.560103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827493/real 1787827493] req@ffff9c1a49d8c000 x1874668972797056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827493.560103:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827493.560179:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972797568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827493.560182:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972797056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827493.560274:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827493.560317:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827493.560319:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827493.560339:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827493.560343:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827493.560353:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972797056:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827493.560406:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3943 00000100:00080000:2.0:1787827493.560414:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827493.560422:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827493.560436:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827493.560457:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972797184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827493.560461:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972797696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827493.560493:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827493.560499:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827493.560503:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827493.560508:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3943 00000100:00080000:2.0:1787827493.560514:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827493.560522:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827493.560538:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827493.560540:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827493.560583:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827493/real 1787827493] req@ffff9c1a49d8e1c0 x1874668972797184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827493.560584:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972797824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827493.560668:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972797184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827493.560686:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827493.560692:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827493.560696:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972797184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827493.560707:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972797312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827493.560736:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827493.560748:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827493/real 1787827493] req@ffff9c1a49d8fc00 x1874668972797312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827493.560761:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972797312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827493.560771:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827493.560774:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827493.560777:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972797312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827493.560784:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972797440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827493.560804:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827493.560816:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827493/real 1787827493] req@ffff9c1a49d8e580 x1874668972797440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827493.560829:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972797440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827493.560839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827493.560842:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827493.560846:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972797440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827493.560852:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972797568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827493.560870:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827493.560881:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827493/real 1787827493] req@ffff9c1a49d8c3c0 x1874668972797568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827493.560894:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972797568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827493.560904:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827493.560907:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827493.560910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972797568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827493.560934:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972797696:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827493.560955:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827493.560966:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827493/real 1787827493] req@ffff9c1a49d8ed00 x1874668972797696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827493.560979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972797696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827493.561007:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827493.561011:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827493.561014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972797696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827493.561021:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972797824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827493.561067:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827493.561079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827493/real 1787827493] req@ffff9c1a49d8cb40 x1874668972797824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827493.561092:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972797824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827493.561102:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827493.561106:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827493.561120:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972797824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827494.071425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827494.071450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827494.327513:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827495.095458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827495.095458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827495.351563:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827496.119514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827496.119518:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827496.375462:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827497.143401:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827497.143419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827497.399520:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827498.167469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827498.167469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827498.423523:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827498.679437:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827498.679444:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827498.679448:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827498.679456:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3949 00000100:00080000:1.0:1787827498.679466:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827498.679472:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827498.679516:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972797952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827498.679546:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827498.679550:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827498.679552:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827498.679556:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3949 00000100:00080000:1.0:1787827498.679560:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827498.679566:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827498.679575:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827498.679584:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972797952:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827498.679589:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972798080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827498.679599:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827498.679601:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827498.679603:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827498.679606:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3949 00000100:00080000:1.0:1787827498.679609:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827498.679611:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827498.679614:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827498.679622:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972798208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827498.679630:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827498.679633:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827498.679634:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827498.679636:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3949 00000100:00080000:1.0:1787827498.679639:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827498.679641:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827498.679644:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827498.679654:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972798336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827498.679655:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827498.679664:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827498.679667:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827498.679668:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827498.679689:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3949 00000100:00080000:1.0:1787827498.679692:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827498.679695:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827498.679699:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827498.679705:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827498/real 1787827498] req@ffff9c1a78a7a940 x1874668972797952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827498.679710:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972798464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827498.679724:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827498.679727:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827498.679730:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827498.679731:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972797952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827498.679735:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3949 00000100:00080000:0.0:1787827498.679752:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827498.679753:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827498.679760:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827498.679760:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827498.679767:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827498.679768:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972797952:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827498.679780:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972798592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827498.679786:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827498.679789:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827498.679790:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827498.679793:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972798080:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827498.679796:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3949 00000100:00080000:1.0:1787827498.679803:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827498.679805:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827498.679809:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827498.679819:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972798720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827498.679825:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827498.679850:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827498/real 1787827498] req@ffff9c1a78a783c0 x1874668972798080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827498.679861:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972798080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827498.679867:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827498.679870:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827498.679872:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972798080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827498.679877:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972798208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827498.679888:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827498.679894:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827498/real 1787827498] req@ffff9c1a78a7a1c0 x1874668972798208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827498.679901:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972798208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827498.679907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827498.679909:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827498.679910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972798208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827498.679914:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972798336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827498.679924:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827498.679930:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827498/real 1787827498] req@ffff9c1a78a78780 x1874668972798336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827498.679938:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972798336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827498.679943:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827498.679944:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827498.679946:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972798336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827498.679949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972798464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827498.679959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827498.679964:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827498/real 1787827498] req@ffff9c1a78a7bc00 x1874668972798464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827498.679975:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972798464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827498.679983:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827498.679987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827498.680001:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972798464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827498.680007:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972798592:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827498.680028:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827498.680038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827498/real 1787827498] req@ffff9c1a78a7b0c0 x1874668972798592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827498.680056:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972798592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827498.680072:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827498.680076:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827498.680079:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972798592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827498.680098:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972798720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827498.680114:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827498.680122:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827498/real 1787827498] req@ffff9c1a78a79e00 x1874668972798720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827498.680129:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972798720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827498.680135:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827498.680137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827498.680139:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972798720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827499.191449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827499.191450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827499.447508:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827500.215470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827500.215470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827500.471577:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827501.239412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827501.239412:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827501.495550:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827502.263465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827502.263490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827502.519552:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827503.287415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827503.287437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827503.543551:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827503.799453:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827503.799461:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827503.799464:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827503.799472:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3954 00000100:00080000:0.0:1787827503.799481:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827503.799487:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827503.799530:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972798848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827503.799555:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827503.799558:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827503.799559:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827503.799562:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3954 00000100:00080000:0.0:1787827503.799565:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827503.799567:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827503.799575:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827503.799585:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972798976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827503.799591:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827503.799594:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827503.799595:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827503.799598:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3954 00000100:00080000:0.0:1787827503.799600:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827503.799604:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827503.799605:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972798848:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787827503.799608:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827503.799617:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972799104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827503.799625:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827503.799627:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827503.799628:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827503.799631:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3954 00000100:00080000:0.0:1787827503.799633:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827503.799636:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827503.799639:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827503.799646:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972799232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827503.799651:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827503.799653:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827503.799655:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827503.799658:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3954 00000100:00080000:0.0:1787827503.799660:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827503.799662:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827503.799665:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827503.799672:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972799360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827503.799677:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827503.799679:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827503.799680:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827503.799683:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3954 00000100:00080000:0.0:1787827503.799684:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827503.799687:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827503.799690:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827503.799698:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972799488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827503.799706:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972798976:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827503.799707:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827503.799714:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827503.799715:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827503.799719:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3954 00000100:00080000:0.0:1787827503.799721:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827503.799724:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827503.799727:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827503.799737:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972799616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827503.799748:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972799104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827503.799766:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972799232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827503.799781:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972799360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827503.799793:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972799488:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827503.799806:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972799616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827504.311430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827504.311471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827504.567452:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827505.335413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827505.335460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827505.591536:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827506.359499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827506.359523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827506.615599:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787827506.871663:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787827506.871715:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827506.871737:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827506.871803:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827506.871830:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827506.871839:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827506.871856:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827506.871867:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787827506.871879:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787827506.871890:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827503/real 1787827506] req@ffff9c1a78860000 x1874668972799104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827506.871923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972799104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827506.871938:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827506.871945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827506.871950:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972799104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827506.871961:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827503/real 1787827506] req@ffff9c1a78863c00 x1874668972799232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827506.871984:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972799232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827506.872016:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827506.872021:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827506.872025:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972799232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827506.872035:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827503/real 1787827506] req@ffff9c1a788630c0 x1874668972799360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827506.872056:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972799360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827506.872066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827506.872070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827506.872074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972799360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827506.872081:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827503/real 1787827506] req@ffff9c1a78861680 x1874668972799488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827506.872094:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972799488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827506.872103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827506.872107:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827506.872110:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972799488:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827506.872117:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827503/real 1787827506] req@ffff9c1a78862580 x1874668972799616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827506.872129:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972799616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827506.872139:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827506.872143:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827506.872147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972799616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827506.872155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827503/real 1787827506] req@ffff9c1a78861e00 x1874668972798848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827506.872167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972798848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827506.872175:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827506.872179:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827506.872183:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972798848:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827506.872208:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827503/real 1787827506] req@ffff9c1a78860780 x1874668972798976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827506.872220:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972798976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827506.872229:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827506.872233:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827506.872236:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972798976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827507.383463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827507.383463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827507.639525:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827508.407482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827508.407483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827508.663571:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827508.919508:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827508.919516:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827508.919519:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827508.919527:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3959 00000100:00080000:1.0:1787827508.919536:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827508.919542:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827508.919570:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972799744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827508.919610:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827508.919614:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827508.919623:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827508.919628:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3959 00000100:00080000:1.0:1787827508.919632:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827508.919637:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827508.919648:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827508.919680:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972799872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827508.919684:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972799744:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827508.919701:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827508.919705:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827508.919707:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827508.919712:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3959 00000100:00080000:1.0:1787827508.919715:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827508.919719:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827508.919724:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827508.919734:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972800000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827508.919746:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787827508.919756:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827508.919765:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827508.919768:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827508.919773:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3959 00000100:00080000:1.0:1787827508.919778:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827508.919782:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827508.919787:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827508.919797:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827508/real 1787827508] req@ffff9c1a78a79e00 x1874668972799744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827508.919803:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972800128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827508.919830:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827508.919834:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827508.919839:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827508.919839:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972799744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827508.919855:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827508.919861:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827508.919867:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3959 00000100:00100000:3.0:1787827508.919869:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972799744:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827508.919874:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827508.919883:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827508.919889:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827508.919900:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972799872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827508.919907:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972800256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827508.919920:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827508.919925:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827508.919927:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827508.919932:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827508.919934:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3959 00000100:00080000:1.0:1787827508.919941:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827508.919946:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827508.919952:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827508.919953:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827508/real 1787827508] req@ffff9c1a78a7b0c0 x1874668972799872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827508.919967:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972800384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827508.919973:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972799872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827508.919989:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827508.919992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827508.919994:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827508.919998:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827508.920000:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827508.920008:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3959 00000100:00100000:3.0:1787827508.920010:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972799872:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827508.920015:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827508.920022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972800000:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827508.920023:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827508.920032:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827508.920049:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972800512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827508.920057:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827508.920078:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827508/real 1787827508] req@ffff9c1a78a7bc00 x1874668972800000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827508.920094:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972800000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827508.920110:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827508.920114:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827508.920118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972800000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827508.920124:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972800128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827508.920145:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827508.920157:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827508/real 1787827508] req@ffff9c1a78a78780 x1874668972800128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827508.920170:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972800128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827508.920197:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827508.920201:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827508.920205:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972800128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827508.920222:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972800256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827508.920244:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827508.920256:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827508/real 1787827508] req@ffff9c1a78a7a1c0 x1874668972800256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827508.920270:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972800256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827508.920408:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827508.920415:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827508.920420:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972800256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827508.920428:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972800384:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827508.920455:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827508.920467:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827508/real 1787827508] req@ffff9c1a78a783c0 x1874668972800384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827508.920478:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972800384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827508.920486:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827508.920489:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827508.920492:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972800384:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827508.920498:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972800512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827508.920515:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827508.920540:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827508/real 1787827508] req@ffff9c1a78a7a940 x1874668972800512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827508.920553:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972800512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827508.920560:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827508.920563:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827508.920566:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972800512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827509.431438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827509.431443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827509.687545:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827510.455455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827510.455459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827510.711531:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827511.479469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827511.479473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827511.735442:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827512.503475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827512.503504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827512.759520:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827513.527456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827513.527456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827513.783494:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827514.039542:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827514.039554:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827514.039560:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827514.039589:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3964 00000100:00080000:3.0:1787827514.039600:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827514.039607:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827514.039669:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972800640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827514.039734:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827514.039741:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827514.039745:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827514.039753:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3964 00000100:00080000:3.0:1787827514.039762:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827514.039770:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827514.039783:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827514.039814:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972800640:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827514.039818:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972800768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827514.039839:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827514.039843:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827514.039845:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827514.039850:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3964 00000100:00080000:3.0:1787827514.039856:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827514.039860:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827514.039875:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827514.039911:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972800896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827514.039951:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827514.039953:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827514.039959:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827514.039963:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827514.039970:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3964 00000100:00080000:3.0:1787827514.039976:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827514.039982:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827514.039988:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827514.040019:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972801024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827514.040029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827514/real 1787827514] req@ffff9c1a7895fc00 x1874668972800640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827514.040031:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827514.040053:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827514.040057:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827514.040083:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3964 00000100:00080000:3.0:1787827514.040088:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827514.040098:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972800640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827514.040098:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827514.040110:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787827514.040128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827514.040134:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972801152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827514.040170:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827514.040182:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972800640:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827514.040196:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827514.040201:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827514.040204:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827514.040211:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3964 00000100:00080000:3.0:1787827514.040217:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827514.040222:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972800768:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827514.040247:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827514.040256:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827514.040279:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972801280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827514.040281:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827514.040301:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827514.040307:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827514/real 1787827514] req@ffff9c1a7895f0c0 x1874668972800768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827514.040309:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827514.040314:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827514.040336:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3964 00000100:00080000:3.0:1787827514.040347:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827514.040375:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827514.040419:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972800768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827514.040448:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827514.040453:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787827514.040458:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827514.040469:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972800768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827514.040481:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972800896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827514.040491:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972801408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827514.040519:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827514.040538:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827514/real 1787827514] req@ffff9c1a7895da40 x1874668972800896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827514.040552:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972800896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827514.040561:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827514.040564:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827514.040567:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972800896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827514.040599:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972801024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827514.040616:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827514.040624:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827514/real 1787827514] req@ffff9c1a7895e1c0 x1874668972801024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827514.040634:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972801024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827514.040642:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827514.040644:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827514.040663:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972801024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827514.040679:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972801152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827514.040699:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827514.040709:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827514/real 1787827514] req@ffff9c1a7895de00 x1874668972801152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827514.040740:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972801152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827514.040749:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827514.040752:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827514.040755:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972801152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827514.040774:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972801280:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827514.040804:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827514.040814:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827514/real 1787827514] req@ffff9c1a7895c780 x1874668972801280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827514.040833:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972801280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827514.040842:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827514.040845:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827514.040848:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972801280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827514.040853:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972801408:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827514.040869:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827514.040878:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827514/real 1787827514] req@ffff9c1a7895cb40 x1874668972801408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827514.040889:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972801408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827514.040908:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827514.040911:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827514.040914:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972801408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827514.551465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827514.551553:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827514.807602:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827515.575455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827515.575457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827515.831508:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827516.599431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827516.599432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827516.855547:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827517.623422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827517.623427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827517.879472:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827518.647493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827518.647493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827518.903447:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827519.159447:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827519.159457:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827519.159462:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827519.159473:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3969 00000100:00080000:3.0:1787827519.159485:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827519.159492:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827519.159535:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972801536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827519.159563:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827519.159567:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827519.159568:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827519.159571:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3969 00000100:00080000:3.0:1787827519.159574:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827519.159577:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827519.159587:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827519.159597:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972801664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827519.159604:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827519.159606:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827519.159607:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827519.159609:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3969 00000100:00080000:3.0:1787827519.159611:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827519.159614:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827519.159617:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827519.159625:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972801792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827519.159632:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827519.159635:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827519.159636:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827519.159638:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3969 00000100:00080000:3.0:1787827519.159640:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827519.159643:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827519.159649:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972801536:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787827519.159649:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827519.159660:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972801920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827519.159666:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827519.159669:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827519.159670:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827519.159673:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3969 00000100:00080000:3.0:1787827519.159676:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827519.159678:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827519.159681:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827519.159688:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972802048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827519.159694:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827519.159697:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827519.159698:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827519.159701:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3969 00000100:00080000:3.0:1787827519.159704:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827519.159706:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827519.159709:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827519.159717:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972802176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827519.159722:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827519.159724:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827519.159726:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827519.159730:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3969 00000800:00000100:0.0:1787827519.159736:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827519.159736:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827519.159743:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827519.159746:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827519.159755:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972802304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827519.159797:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827519/real 1787827519] req@ffff9c1a7895d2c0 x1874668972801536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827519.159815:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972801536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827519.159824:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827519.159828:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827519.159831:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972801536:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827519.159837:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972801664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827519.159857:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827519.159866:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827519/real 1787827519] req@ffff9c1a7895cb40 x1874668972801664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827519.159875:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972801664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827519.159881:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827519.159883:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827519.159885:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972801664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827519.159888:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972801792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827519.159898:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827519.159904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827519/real 1787827519] req@ffff9c1a7895c780 x1874668972801792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827519.159916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972801792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827519.159921:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827519.159923:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827519.159925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972801792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827519.159946:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972801920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827519.159967:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827519.159975:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827519/real 1787827519] req@ffff9c1a7895de00 x1874668972801920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827519.159983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972801920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827519.159989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827519.159991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827519.159993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972801920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827519.159996:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972802048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827519.160007:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827519.160013:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827519/real 1787827519] req@ffff9c1a7895e1c0 x1874668972802048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827519.160020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972802048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827519.160030:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827519.160032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827519.160034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972802048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827519.160037:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972802176:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827519.160048:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827519.160053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827519/real 1787827519] req@ffff9c1a7895da40 x1874668972802176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827519.160060:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972802176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827519.160065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827519.160067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827519.160069:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972802176:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827519.160072:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972802304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827519.160082:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827519.160098:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827519/real 1787827519] req@ffff9c1a7895f0c0 x1874668972802304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827519.160125:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972802304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827519.160134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827519.160137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827519.160142:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972802304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827519.671451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827519.671455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827519.927499:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827520.695418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827520.695457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827520.951554:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827521.719516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827521.719516:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827521.975525:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827522.743472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827522.743483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827522.999535:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827523.767462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827523.767466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827524.023585:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827524.279446:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827524.279454:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827524.279463:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827524.279471:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3974 00000100:00080000:3.0:1787827524.279481:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827524.279486:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827524.279528:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972802432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827524.279560:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827524.279564:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827524.279566:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827524.279571:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3974 00000100:00080000:3.0:1787827524.279574:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827524.279590:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827524.279600:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827524.279628:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972802560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827524.279640:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827524.279654:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827524.279657:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827524.279662:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3974 00000100:00100000:0.0:1787827524.279666:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972802432:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827524.279668:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827524.279678:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827524.279684:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827524.279698:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972802688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827524.279720:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827524.279724:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827524.279726:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827524.279730:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3974 00000100:00080000:3.0:1787827524.279735:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827524.279737:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827524.279744:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827524.279751:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827524.279766:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972802816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827524.279785:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827524/real 1787827524] req@ffff9c1a7895fc00 x1874668972802432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827524.279792:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827524.279799:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827524.279801:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827524.279804:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972802432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827524.279811:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3974 00000100:00080000:0.0:1787827524.279817:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827524.279819:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827524.279823:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827524.279830:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972802432:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827524.279831:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827524.279840:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972802560:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787827524.279841:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827524.279864:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972802944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827524.279867:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827524.279880:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827524.279884:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787827524.279888:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827524/real 1787827524] req@ffff9c1a7895e580 x1874668972802560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827524.279889:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827524.279898:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3974 00000100:00080000:3.0:1787827524.279904:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827524.279910:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827524.279915:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827524.279918:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972802560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827524.279928:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972803072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827524.279937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827524.279940:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827524.279944:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827524.279945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827524.279947:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827524.279951:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972802560:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827524.279954:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3974 00000100:00080000:3.0:1787827524.279960:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827524.279963:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827524.279966:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827524.279983:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972803200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827524.279991:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972802688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827524.280012:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827524.280023:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827524/real 1787827524] req@ffff9c1a7895f840 x1874668972802688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827524.280032:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972802688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827524.280038:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827524.280041:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827524.280043:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972802688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827524.280048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972802816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827524.280062:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827524.280092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827524/real 1787827524] req@ffff9c1a7895c000 x1874668972802816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827524.280101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972802816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827524.280107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827524.280110:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827524.280113:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972802816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827524.280128:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972802944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827524.280147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827524.280154:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827524/real 1787827524] req@ffff9c1a7895f480 x1874668972802944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827524.280162:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972802944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827524.280168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827524.280170:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827524.280172:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972802944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827524.280177:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972803072:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827524.280194:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827524.280204:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827524/real 1787827524] req@ffff9c1a7895cf00 x1874668972803072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827524.280216:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972803072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827524.280222:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827524.280224:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827524.280227:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972803072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827524.280242:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972803200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827524.280254:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827524.280266:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827524/real 1787827524] req@ffff9c1a7895c3c0 x1874668972803200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827524.280275:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972803200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827524.280293:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827524.280296:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827524.280298:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972803200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827524.791414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827524.791443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827525.047522:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827525.815529:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827525.815530:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827526.071569:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827526.839405:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827526.839413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827527.095542:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827527.863478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827527.863478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827528.119601:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827528.887462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827528.887479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827529.143512:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827529.399479:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827529.399488:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827529.399492:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827529.399501:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3979 00000100:00080000:1.0:1787827529.399511:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827529.399517:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827529.399550:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972803328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827529.399573:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827529.399576:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827529.399578:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827529.399581:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3979 00000100:00080000:1.0:1787827529.399583:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827529.399586:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827529.399594:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827529.399604:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972803456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827529.399613:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827529.399618:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787827529.399620:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972803328:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827529.399621:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827529.399629:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3979 00000100:00080000:1.0:1787827529.399632:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827529.399635:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827529.399638:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827529.399650:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972803584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827529.399666:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827529.399668:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827529.399670:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827529.399673:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3979 00000100:00080000:1.0:1787827529.399676:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827529.399679:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827529.399682:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827529.399691:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972803712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827529.399698:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827529.399701:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827529.399702:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827529.399709:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3979 00000800:00000100:0.0:1787827529.399710:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827529.399714:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827529.399718:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827529.399724:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827529.399734:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972803840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827529.399744:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827529.399747:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827529.399750:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827529.399754:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3979 00000100:00080000:1.0:1787827529.399758:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827529.399759:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827529/real 1787827529] req@ffff9c1a78a78f00 x1874668972803328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827529.399763:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827529.399771:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827529.399785:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972803968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827529.399791:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972803328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827529.399798:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827529.399806:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827529.399810:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827529.399815:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3979 00000100:00080000:1.0:1787827529.399817:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827529.399819:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827529.399822:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827529.399824:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827529.399835:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827529.399838:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972804096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827529.399845:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972803328:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827529.399876:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972803456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827529.399897:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827529.399908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827529/real 1787827529] req@ffff9c1a78a7a580 x1874668972803456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827529.399918:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972803456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827529.399924:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827529.399927:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827529.399930:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972803456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827529.399934:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972803584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827529.399953:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827529.399961:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827529/real 1787827529] req@ffff9c1a78a78000 x1874668972803584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827529.399969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972803584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827529.399975:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827529.399978:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827529.399980:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972803584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827529.399984:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972803712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827529.399995:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827529.400002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827529/real 1787827529] req@ffff9c1a78a7b480 x1874668972803712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827529.400010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972803712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827529.400016:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827529.400018:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827529.400020:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972803712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827529.400023:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972803840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827529.400051:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827529.400059:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827529/real 1787827529] req@ffff9c1a78a792c0 x1874668972803840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827529.400068:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972803840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827529.400074:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827529.400076:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827529.400079:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972803840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827529.400083:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972803968:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827529.400095:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827529.400102:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827529/real 1787827529] req@ffff9c1a78a7b840 x1874668972803968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827529.400110:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972803968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827529.400116:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827529.400118:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827529.400120:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972803968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827529.400124:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972804096:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827529.400135:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827529.400141:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827529/real 1787827529] req@ffff9c1a78a79a40 x1874668972804096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827529.400155:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972804096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827529.400160:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827529.400162:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827529.400164:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972804096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827529.911508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827529.911510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827530.167597:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827530.935524:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827530.935539:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827531.191511:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827531.959489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827531.959489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827532.215780:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827532.983472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827532.983501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827533.239434:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827534.007465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827534.007465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827534.263595:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827534.519504:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827534.519515:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827534.519526:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827534.519537:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3984 00000100:00080000:0.0:1787827534.519551:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827534.519558:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827534.519603:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972804224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827534.519634:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827534.519637:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827534.519638:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827534.519642:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3984 00000100:00080000:0.0:1787827534.519644:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827534.519648:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827534.519659:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827534.519669:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972804352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827534.519677:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827534.519680:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827534.519682:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827534.519684:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3984 00000100:00080000:0.0:1787827534.519686:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827534.519692:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827534.519693:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972804224:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787827534.519696:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827534.519708:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972804480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827534.519723:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827534.519725:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827534.519727:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827534.519730:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3984 00000100:00080000:0.0:1787827534.519732:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827534.519735:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827534.519738:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827534.519749:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972804608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827534.519755:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827534.519759:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827534.519761:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827534.519763:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3984 00000100:00080000:0.0:1787827534.519766:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827534.519768:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827534.519772:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827534.519774:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827534.519783:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972804736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827534.519794:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827534.519798:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827534.519800:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827534.519822:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3984 00000100:00080000:0.0:1787827534.519827:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827534.519833:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827534.519840:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827534.519856:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827534/real 1787827534] req@ffff9c1a788621c0 x1874668972804224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827534.519860:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972804864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827534.519884:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827534.519887:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827534.519889:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827534.519892:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972804224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827534.519896:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3984 00000100:00080000:0.0:1787827534.519904:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827534.519909:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787827534.519911:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827534.519918:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827534.519919:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827534.519926:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972804224:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827534.519934:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972804992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827534.519936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972804352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827534.519965:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827534.519977:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827534/real 1787827534] req@ffff9c1a78862940 x1874668972804352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827534.519986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972804352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827534.519993:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827534.520012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827534.520016:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972804352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827534.520044:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972804480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827534.520061:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827534.520070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827534/real 1787827534] req@ffff9c1a78860b40 x1874668972804480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827534.520078:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972804480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827534.520085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827534.520087:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827534.520090:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972804480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827534.520094:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972804608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827534.520104:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827534.520110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827534/real 1787827534] req@ffff9c1a78861a40 x1874668972804608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827534.520118:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972804608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827534.520124:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827534.520126:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827534.520128:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972804608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827534.520132:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972804736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827534.520144:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827534.520151:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827534/real 1787827534] req@ffff9c1a78860f00 x1874668972804736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827534.520160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972804736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827534.520165:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827534.520167:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827534.520169:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972804736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827534.520173:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972804864:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827534.520185:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827534.520191:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827534/real 1787827534] req@ffff9c1a78862d00 x1874668972804864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827534.520199:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972804864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827534.520205:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827534.520207:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827534.520209:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972804864:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827534.520213:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972804992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827534.520223:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827534.520229:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827534/real 1787827534] req@ffff9c1a788612c0 x1874668972804992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827534.520237:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972804992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827534.520248:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827534.520250:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827534.520252:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972804992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827535.031485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827535.031485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827535.287538:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827536.055428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827536.055432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827536.311540:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827537.079507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827537.079508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827537.335584:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827538.103453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827538.103456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827538.359590:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827539.127454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827539.127454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827539.383642:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827539.639528:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827539.639540:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827539.639546:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827539.639558:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3989 00000100:00080000:2.0:1787827539.639571:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827539.639581:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827539.639642:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972805120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827539.639686:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827539.639691:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827539.639694:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827539.639699:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3989 00000100:00080000:2.0:1787827539.639703:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827539.639708:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827539.639720:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827539.639737:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972805248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827539.639752:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827539.639757:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827539.639759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827539.639763:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3989 00000100:00080000:2.0:1787827539.639768:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827539.639772:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827539.639778:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827539.639789:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972805120:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827539.639793:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972805376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827539.639811:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827539.639820:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827539.639823:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827539.639827:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3989 00000100:00080000:2.0:1787827539.639830:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827539.639834:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827539.639840:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827539.639853:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972805504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827539.639865:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827539.639869:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827539.639872:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827539.639877:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3989 00000100:00080000:2.0:1787827539.639894:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827539.639900:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787827539.639910:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787827539.639914:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827539.639933:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972805632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827539.639945:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827539.639949:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827539.639951:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827539.639955:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3989 00000100:00080000:2.0:1787827539.639981:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827539.640001:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827539.640011:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827539.640019:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827539/real 1787827539] req@ffff9c1a49d8cb40 x1874668972805120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827539.640039:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972805760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827539.640072:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972805120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827539.640090:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827539.640101:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827539.640105:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827539.640107:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827539.640128:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827539.640130:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3989 00000100:00080000:2.0:1787827539.640144:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827539.640153:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972805120:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827539.640157:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827539.640174:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827539.640188:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972805248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827539.640199:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972805888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827539.640266:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827539.640297:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827539/real 1787827539] req@ffff9c1a49d8ed00 x1874668972805248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827539.640316:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972805248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827539.640327:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827539.640332:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827539.640337:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972805248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827539.640429:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972805376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827539.640463:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827539.640485:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827539/real 1787827539] req@ffff9c1a49d8c3c0 x1874668972805376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827539.640521:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972805376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827539.640540:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827539.640546:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827539.640554:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972805376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827539.640568:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972805504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827539.640609:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827539.640632:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827539/real 1787827539] req@ffff9c1a49d8e580 x1874668972805504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827539.640658:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972805504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827539.640677:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827539.640683:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827539.640689:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972805504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827539.640701:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972805632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827539.640740:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827539.640764:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827539/real 1787827539] req@ffff9c1a49d8fc00 x1874668972805632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827539.640787:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972805632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827539.640798:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827539.640802:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827539.640806:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972805632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827539.640814:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972805760:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827539.640842:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827539.640855:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827539/real 1787827539] req@ffff9c1a49d8e1c0 x1874668972805760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827539.640870:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972805760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827539.640880:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827539.640883:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827539.640887:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972805760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827539.640893:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972805888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827539.640911:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827539.640921:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827539/real 1787827539] req@ffff9c1a49d8c000 x1874668972805888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827539.640944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972805888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827539.640989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827539.641006:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827539.641013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972805888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827540.151450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827540.151584:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827540.407498:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827541.175466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827541.175466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827541.431570:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827542.199461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827542.199483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827542.455474:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827543.223523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827543.223529:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827543.479564:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827544.247430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827544.247477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827544.503504:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827544.759465:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827544.759473:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827544.759476:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827544.759485:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 3995 00000100:00080000:3.0:1787827544.759493:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827544.759498:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827544.759530:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972806016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827544.759556:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827544.759559:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827544.759561:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827544.759564:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3995 00000100:00080000:3.0:1787827544.759566:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827544.759569:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827544.759577:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827544.759591:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972806144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827544.759602:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827544.759606:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827544.759609:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827544.759613:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 3995 00000100:00080000:3.0:1787827544.759618:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827544.759622:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827544.759628:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827544.759629:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972806016:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827544.759642:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972806272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827544.759656:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827544.759660:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827544.759662:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827544.759667:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3995 00000100:00080000:3.0:1787827544.759671:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827544.759677:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827544.759682:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827544.759687:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827544.759697:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972806400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827544.759705:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827544.759708:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827544.759709:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827544.759712:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3995 00000100:00080000:3.0:1787827544.759715:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827544.759718:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827544.759723:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827544.759724:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827544/real 1787827544] req@ffff9c1a7895e940 x1874668972806016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827544.759734:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972806528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827544.759742:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827544.759743:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972806016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827544.759746:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827544.759750:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827544.759759:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3995 00000100:00080000:3.0:1787827544.759762:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827544.759763:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827544.759766:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787827544.759770:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787827544.759772:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827544.759776:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972806016:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827544.759782:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972806656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827544.759785:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972806144:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827544.759797:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827544.759800:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827544.759802:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827544.759805:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 3995 00000100:00080000:3.0:1787827544.759808:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827544.759811:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827544.759815:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827544.759820:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827544.759826:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827544/real 1787827544] req@ffff9c1a7895c3c0 x1874668972806144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827544.759830:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972806784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827544.759849:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972806144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827544.759868:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827544.759871:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827544.759874:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972806144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827544.759897:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972806272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827544.759919:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827544.759946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827544/real 1787827544] req@ffff9c1a7895cf00 x1874668972806272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827544.759960:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972806272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827544.759968:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827544.759972:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827544.759974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972806272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827544.759978:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972806400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827544.759994:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827544.760001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827544/real 1787827544] req@ffff9c1a7895f480 x1874668972806400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827544.760009:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972806400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827544.760014:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827544.760016:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827544.760018:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972806400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827544.760021:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972806528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827544.760031:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827544.760037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827544/real 1787827544] req@ffff9c1a7895c000 x1874668972806528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827544.760044:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972806528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827544.760049:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827544.760051:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827544.760053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972806528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827544.760056:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972806656:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827544.760066:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827544.760072:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827544/real 1787827544] req@ffff9c1a7895f840 x1874668972806656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827544.760079:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972806656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827544.760084:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827544.760085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827544.760087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972806656:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827544.760090:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972806784:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827544.760100:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827544.760105:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827544/real 1787827544] req@ffff9c1a7895e580 x1874668972806784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827544.760112:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972806784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827544.760117:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827544.760119:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827544.760121:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972806784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827545.271521:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827545.271541:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827545.527541:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827546.295430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827546.295484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827546.551481:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827547.319455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827547.319501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827547.575470:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827548.343419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827548.343488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827548.599490:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827549.367479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827549.367498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827549.623530:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827549.879437:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827549.879448:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827549.879453:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827549.879464:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4000 00000100:00080000:3.0:1787827549.879476:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827549.879483:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827549.879536:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972806912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827549.879561:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827549.879565:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827549.879567:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827549.879589:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4000 00000100:00080000:3.0:1787827549.879596:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827549.879604:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827549.879619:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827549.879656:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972806912:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827549.879658:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972807040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827549.879673:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827549.879676:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827549.879677:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827549.879681:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4000 00000100:00080000:3.0:1787827549.879685:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827549.879688:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827549.879692:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827549.879703:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972807168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827549.879736:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827549.879739:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827549.879745:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827549.879748:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827549.879755:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4000 00000100:00080000:3.0:1787827549.879760:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827549.879765:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827549.879771:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827549.879802:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972807296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827549.879814:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827549/real 1787827549] req@ffff9c1a7895fc00 x1874668972806912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827549.879818:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827549.879831:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827549.879838:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827549.879850:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4000 00000100:00080000:3.0:1787827549.879858:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827549.879860:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972806912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827549.879871:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787827549.879882:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827549.879906:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787827549.879939:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827549.879944:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972806912:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827549.879955:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972807424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827549.879967:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827549.879969:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972807040:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827549.879971:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827549.879986:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827549.879990:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4000 00000100:00080000:3.0:1787827549.879994:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827549.879999:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787827549.880004:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787827549.880004:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827549.880020:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972807552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827549.880021:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827549/real 1787827549] req@ffff9c1a7895f0c0 x1874668972807040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827549.880054:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827549.880061:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827549.880065:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827549.880072:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4000 00000100:00080000:3.0:1787827549.880074:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827549.880076:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972807040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827549.880080:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827549.880087:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827549.880089:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827549.880095:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827549.880097:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972807680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827549.880100:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972807040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827549.880131:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972807168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827549.880162:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827549.880174:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827549/real 1787827549] req@ffff9c1a7895da40 x1874668972807168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827549.880187:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972807168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827549.880195:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827549.880206:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827549.880209:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972807168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827549.880215:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972807296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827549.880231:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827549.880240:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827549/real 1787827549] req@ffff9c1a7895e1c0 x1874668972807296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827549.880252:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972807296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827549.880268:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827549.880271:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827549.880285:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972807296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827549.880300:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972807424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827549.880316:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827549.880386:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827549/real 1787827549] req@ffff9c1a7895de00 x1874668972807424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827549.880403:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972807424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827549.880411:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827549.880415:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827549.880418:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972807424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827549.880425:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972807552:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827549.880449:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827549.880460:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827549/real 1787827549] req@ffff9c1a7895c780 x1874668972807552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827549.880472:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972807552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827549.880480:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827549.880482:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827549.880486:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972807552:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827549.880506:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972807680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827549.880517:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827549.880523:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827549/real 1787827549] req@ffff9c1a7895cb40 x1874668972807680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827549.880530:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972807680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827549.880535:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827549.880537:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827549.880539:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972807680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827550.391483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827550.391661:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827550.647512:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827551.415486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827551.415486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827551.671539:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827552.439479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827552.439479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827552.695470:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827553.463435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827553.463469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827553.719537:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827554.487462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827554.487462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827554.743509:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827554.999471:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827554.999480:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827554.999484:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827554.999494:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4005 00000100:00080000:1.0:1787827554.999506:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827554.999513:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827554.999555:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972807808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827554.999588:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827554.999590:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827554.999592:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827554.999596:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4005 00000100:00080000:1.0:1787827554.999599:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827554.999603:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827554.999611:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827554.999624:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972807936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827554.999629:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972807808:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827554.999636:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827554.999640:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827554.999641:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827554.999645:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4005 00000100:00080000:1.0:1787827554.999647:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827554.999649:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827554.999654:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827554.999667:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972808064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827554.999679:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827554.999683:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827554.999686:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827554.999690:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827554.999695:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4005 00000100:00080000:1.0:1787827554.999700:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827554.999705:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827554.999710:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827554.999721:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972808192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827554.999728:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827554.999730:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787827554.999733:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827554/real 1787827554] req@ffff9c1a78a79a40 x1874668972807808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827554.999733:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827554.999740:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4005 00000100:00080000:1.0:1787827554.999746:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827554.999752:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827554.999756:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827554.999760:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972807808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827554.999767:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972808320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827554.999772:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827554.999781:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827554.999786:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972807808:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827554.999789:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827554.999793:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827554.999795:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827554.999800:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4005 00000100:00080000:1.0:1787827554.999805:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827554.999807:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972807936:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827554.999813:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827554.999820:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827554.999830:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827554.999837:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972808448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827554.999843:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827554/real 1787827554] req@ffff9c1a78a7b840 x1874668972807936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827554.999877:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827554.999882:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972807936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827554.999883:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827554.999887:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827554.999897:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827554.999897:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4005 00000100:00080000:0.0:1787827554.999905:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827554.999905:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827554.999912:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972807936:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827554.999915:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827554.999920:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972808064:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787827554.999922:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827554.999938:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972808576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827554.999945:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827554.999958:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827554/real 1787827554] req@ffff9c1a78a792c0 x1874668972808064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827554.999967:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972808064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827554.999973:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827554.999975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827554.999977:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972808064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827554.999981:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972808192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827554.999991:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827554.999998:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827554/real 1787827554] req@ffff9c1a78a7b480 x1874668972808192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827555.000005:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972808192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827555.000010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827555.000012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827555.000014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972808192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827555.000017:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972808320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827555.000026:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827555.000032:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827554/real 1787827555] req@ffff9c1a78a78000 x1874668972808320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827555.000039:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972808320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827555.000044:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827555.000045:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827555.000047:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972808320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827555.000058:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972808448:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827555.000068:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827555.000073:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827554/real 1787827555] req@ffff9c1a78a7a580 x1874668972808448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827555.000080:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972808448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827555.000086:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827555.000088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827555.000090:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972808448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827555.000093:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972808576:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827555.000101:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827555.000106:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827554/real 1787827555] req@ffff9c1a78a78f00 x1874668972808576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827555.000113:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972808576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827555.000118:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827555.000120:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827555.000122:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972808576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827555.511417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827555.511425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827555.767481:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827556.535515:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827556.535516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827556.791553:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827557.559421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827557.559421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827557.815526:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827558.583475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827558.583475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827558.839515:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827559.607409:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827559.607462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827559.863539:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827560.119507:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827560.119523:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827560.119528:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827560.119538:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4010 00000100:00080000:0.0:1787827560.119549:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827560.119557:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827560.119589:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972808704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827560.119615:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827560.119619:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827560.119622:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827560.119627:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4010 00000100:00080000:0.0:1787827560.119631:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827560.119636:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827560.119646:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827560.119667:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972808832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827560.119667:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972808704:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827560.119688:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827560.119691:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827560.119693:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827560.119697:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4010 00000100:00080000:0.0:1787827560.119700:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827560.119703:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827560.119708:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827560.119720:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972808960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827560.119728:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827560.119731:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827560.119733:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827560.119740:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4010 00000800:00000100:1.0:1787827560.119742:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827560.119745:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827560.119750:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827560.119754:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827560.119765:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972809088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827560.119772:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827560.119775:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827560.119784:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827560.119788:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827560/real 1787827560] req@ffff9c1a78863840 x1874668972808704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827560.119789:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4010 00000100:00080000:0.0:1787827560.119794:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827560.119800:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827560.119806:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827560.119836:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972809216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827560.119838:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972808704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827560.119853:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827560.119861:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827560.119862:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827560.119865:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827560.119866:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4010 00000100:00080000:0.0:1787827560.119873:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827560.119875:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827560.119879:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827560.119882:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972808704:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787827560.119885:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827560.119897:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972809344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827560.119906:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827560.119909:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972808832:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827560.119910:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827560.119914:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827560.119919:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4010 00000100:00080000:0.0:1787827560.119922:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827560.119925:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827560.119929:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827560.119937:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972809472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827560.119943:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827560.119962:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827560/real 1787827560] req@ffff9c1a788612c0 x1874668972808832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827560.119972:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972808832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827560.119978:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827560.119981:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827560.119984:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972808832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827560.119988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972808960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827560.120000:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827560.120007:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827560/real 1787827560] req@ffff9c1a78862d00 x1874668972808960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827560.120014:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972808960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827560.120024:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827560.120027:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827560.120029:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972808960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827560.120032:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972809088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827560.120043:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827560.120049:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827560/real 1787827560] req@ffff9c1a78860f00 x1874668972809088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827560.120057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972809088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827560.120064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827560.120066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827560.120069:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972809088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827560.120072:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972809216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827560.120083:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827560.120092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827560/real 1787827560] req@ffff9c1a78861a40 x1874668972809216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827560.120100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972809216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827560.120106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827560.120109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827560.120112:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972809216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827560.120130:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972809344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827560.120144:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827560.120155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827560/real 1787827560] req@ffff9c1a78860b40 x1874668972809344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827560.120167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972809344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827560.120173:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827560.120176:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827560.120178:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972809344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827560.120181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972809472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827560.120192:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827560.120200:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827560/real 1787827560] req@ffff9c1a78862940 x1874668972809472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827560.120207:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972809472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827560.120213:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827560.120215:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827560.120217:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972809472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827560.631436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827560.631436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827560.887501:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827561.655422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827561.655433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827561.911479:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827562.679469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827562.679469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827562.935567:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827563.703442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827563.703572:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827563.959556:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827564.727429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827564.727429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827564.983537:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827565.239460:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827565.239470:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827565.239486:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827565.239495:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4015 00000100:00080000:3.0:1787827565.239505:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827565.239512:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827565.239583:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972809600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827565.239619:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827565.239624:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827565.239626:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827565.239631:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4015 00000100:00080000:3.0:1787827565.239646:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827565.239650:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827565.239661:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827565.239677:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972809728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827565.239690:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827565.239695:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827565.239697:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827565.239712:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4015 00000100:00080000:3.0:1787827565.239716:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827565.239727:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827565.239729:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972809600:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787827565.239734:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827565.239754:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972809856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827565.239777:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827565.239820:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827565.239823:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827565.239840:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4015 00000100:00080000:3.0:1787827565.239847:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827565.239853:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827565.239861:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827565.239887:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827565.239906:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972809984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827565.239928:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827565.239934:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827565.239938:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827565.239946:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4015 00000100:00100000:1.0:1787827565.239954:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827565/real 1787827565] req@ffff9c1a7895d2c0 x1874668972809600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827565.239957:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827565.239972:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827565.240016:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827565.240023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972809600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827565.240034:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972810112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827565.240050:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827565.240051:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827565.240060:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827565.240063:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827565.240066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827565.240070:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4015 00000100:00080000:3.0:1787827565.240077:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827565.240083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972809600:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827565.240086:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827565.240097:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827565.240104:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972809728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827565.240109:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972810240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827565.240132:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827565.240135:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827565.240137:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827565.240140:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4015 00000100:00080000:3.0:1787827565.240143:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827565.240146:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827565.240151:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827565.240165:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972810368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827565.240166:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827565.240205:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827565/real 1787827565] req@ffff9c1a7895cb40 x1874668972809728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827565.240226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972809728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827565.240237:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827565.240242:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827565.240247:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972809728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827565.240276:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972809856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827565.240310:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827565.240323:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827565/real 1787827565] req@ffff9c1a7895c780 x1874668972809856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827565.240335:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972809856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827565.240342:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827565.240344:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827565.240347:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972809856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827565.240352:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972809984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827565.240368:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827565.240379:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827565/real 1787827565] req@ffff9c1a7895de00 x1874668972809984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827565.240423:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972809984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827565.240432:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827565.240434:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827565.240438:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972809984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827565.240443:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972810112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827565.240470:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827565.240479:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827565/real 1787827565] req@ffff9c1a7895e1c0 x1874668972810112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827565.240495:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972810112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827565.240502:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827565.240504:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827565.240506:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972810112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827565.240522:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972810240:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827565.240536:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827565.240543:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827565/real 1787827565] req@ffff9c1a7895da40 x1874668972810240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827565.240553:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972810240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827565.240559:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827565.240562:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827565.240564:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972810240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827565.240568:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972810368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827565.240580:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827565.240587:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827565/real 1787827565] req@ffff9c1a7895f0c0 x1874668972810368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827565.240607:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972810368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827565.240618:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827565.240621:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827565.240623:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972810368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827565.751445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827565.751519:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827566.007495:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827566.775451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827566.775465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827567.031706:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827567.799510:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827567.799541:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827568.055516:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827568.823486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827568.823549:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827569.079518:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827569.847515:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827569.847530:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827570.103536:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827570.359440:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827570.359449:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827570.359453:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827570.359462:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4020 00000100:00080000:3.0:1787827570.359473:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827570.359479:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827570.359529:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972810496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827570.359569:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827570.359572:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827570.359574:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827570.359577:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4020 00000100:00080000:3.0:1787827570.359581:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827570.359584:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827570.359592:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827570.359604:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972810624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827570.359612:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827570.359614:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827570.359616:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827570.359619:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4020 00000100:00080000:3.0:1787827570.359621:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827570.359623:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827570.359627:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827570.359634:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972810752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827570.359641:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972810496:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827570.359646:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827570.359650:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827570.359652:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827570.359655:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4020 00000100:00080000:3.0:1787827570.359657:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827570.359659:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827570.359664:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827570.359672:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972810880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827570.359679:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827570.359681:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827570.359683:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827570.359685:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4020 00000100:00080000:3.0:1787827570.359687:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827570.359690:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827570.359693:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827570.359701:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972811008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827570.359707:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827570.359709:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827570.359710:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827570.359713:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4020 00000100:00080000:3.0:1787827570.359716:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827570.359719:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827570.359722:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827570.359731:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972811136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827570.359740:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972810624:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827570.359741:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827570.359758:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827570.359760:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827570.359769:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4020 00000100:00080000:3.0:1787827570.359772:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827570.359776:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827570.359781:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827570.359803:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972811264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827570.359820:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972810752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827570.359842:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972810880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827570.359861:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972811008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827570.359874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972811136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827570.359887:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972811264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827570.871520:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827570.871533:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827571.127568:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827571.895469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827571.895469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827572.151554:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827572.919496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827572.919517:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827573.175560:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827573.431621:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787827573.431676:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827573.431693:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827573.431750:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827573.431759:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827573.431769:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827573.431786:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827573.431795:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827573.431805:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787827573.431853:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827570/real 1787827573] req@ffff9c1a7895f840 x1874668972810752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827573.431892:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972810752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827573.431905:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827573.431910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827573.431914:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972810752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827573.431922:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827570/real 1787827573] req@ffff9c1a7895c000 x1874668972810880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827573.431937:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972810880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827573.431948:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827573.431953:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827573.431957:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972810880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827573.431967:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827570/real 1787827573] req@ffff9c1a7895f480 x1874668972811008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827573.431981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972811008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827573.431990:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827573.431994:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827573.431998:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972811008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827573.432006:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827570/real 1787827573] req@ffff9c1a7895cf00 x1874668972811136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827573.432019:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972811136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827573.432029:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827573.432033:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827573.432037:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972811136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827573.432044:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827570/real 1787827573] req@ffff9c1a7895c3c0 x1874668972811264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827573.432056:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972811264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827573.432064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827573.432068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827573.432071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972811264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827573.432079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827570/real 1787827573] req@ffff9c1a7895fc00 x1874668972810496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827573.432094:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972810496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827573.432103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827573.432107:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827573.432111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972810496:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827573.432120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827570/real 1787827573] req@ffff9c1a7895e580 x1874668972810624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827573.432136:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972810624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827573.432147:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827573.432151:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827573.432155:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972810624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827573.943464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827573.943496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827574.199637:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827574.967441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827574.967441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827575.223534:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827575.479448:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827575.479456:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827575.479460:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827575.479469:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4025 00000100:00080000:3.0:1787827575.479478:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827575.479484:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827575.479513:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972811392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827575.479535:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827575.479538:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827575.479539:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827575.479542:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4025 00000100:00080000:3.0:1787827575.479544:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827575.479548:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827575.479561:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827575.479571:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972811520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827575.479579:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827575.479581:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827575.479582:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827575.479585:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4025 00000100:00080000:3.0:1787827575.479587:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827575.479590:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827575.479594:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827575.479598:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972811392:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827575.479604:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972811648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827575.479613:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827575.479616:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827575.479616:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827575.479619:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4025 00000100:00080000:3.0:1787827575.479621:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827575.479623:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827575.479626:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827575.479634:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972811776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827575.479640:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827575.479642:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827575.479644:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827575.479646:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4025 00000100:00080000:3.0:1787827575.479649:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827575.479651:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827575.479654:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827575.479662:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972811904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827575.479669:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827575.479670:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827575.479672:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827575.479675:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827575.479679:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4025 00000100:00080000:3.0:1787827575.479681:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827575.479684:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827575.479688:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827575.479694:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972812032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827575.479700:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827575.479702:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827575.479704:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827575.479722:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4025 00000100:00100000:1.0:1787827575.479723:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827575/real 1787827575] req@ffff9c1a7895e940 x1874668972811392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827575.479728:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827575.479736:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827575.479741:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827575.479750:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972811392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827575.479753:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972812160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827575.479786:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827575.479793:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827575.479797:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972811392:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827575.479813:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972811520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827575.479840:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827575.479853:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827575/real 1787827575] req@ffff9c1a7895e580 x1874668972811520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827575.479866:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972811520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827575.479875:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827575.479894:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827575.479898:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972811520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827575.479926:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972811648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827575.479949:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827575.479961:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827575/real 1787827575] req@ffff9c1a7895fc00 x1874668972811648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827575.479973:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972811648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827575.479981:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827575.479984:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827575.479987:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972811648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827575.479994:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972811776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827575.480015:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827575.480027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827575/real 1787827575] req@ffff9c1a7895c3c0 x1874668972811776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827575.480040:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972811776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827575.480049:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827575.480052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827575.480055:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972811776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827575.480060:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972811904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827575.480079:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827575.480090:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827575/real 1787827575] req@ffff9c1a7895cf00 x1874668972811904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827575.480102:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972811904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827575.480109:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827575.480119:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827575.480122:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972811904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827575.480128:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972812032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827575.480150:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827575.480162:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827575/real 1787827575] req@ffff9c1a7895f480 x1874668972812032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827575.480174:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972812032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827575.480183:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827575.480186:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827575.480189:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972812032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827575.480196:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972812160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827575.480215:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827575.480233:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827575/real 1787827575] req@ffff9c1a7895c000 x1874668972812160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827575.480247:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972812160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827575.480262:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827575.480265:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827575.480269:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972812160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827575.991444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827575.991445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827576.247524:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827577.015413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827577.015423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827577.271514:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827578.039447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827578.039447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827578.295560:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827579.063413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827579.063428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827579.319502:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827580.087478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827580.087478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827580.343514:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827580.599554:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827580.599562:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827580.599565:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827580.599573:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4030 00000100:00080000:0.0:1787827580.599582:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827580.599588:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827580.599626:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972812288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827580.599646:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827580.599649:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827580.599650:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827580.599653:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4030 00000100:00080000:0.0:1787827580.599656:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827580.599659:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827580.599666:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827580.599689:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972812416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827580.599698:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827580.599699:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972812288:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827580.599703:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827580.599707:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827580.599727:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4030 00000100:00080000:0.0:1787827580.599730:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827580.599733:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827580.599737:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827580.599748:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972812544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827580.599757:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827580.599760:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827580.599761:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827580.599763:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4030 00000100:00080000:0.0:1787827580.599767:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827580.599769:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827580.599772:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827580.599778:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827580.599782:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972812672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827580.599792:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827580.599795:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827580.599796:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827580.599799:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4030 00000100:00080000:0.0:1787827580.599802:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827580.599805:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827580.599809:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827580.599814:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827580/real 1787827580] req@ffff9c1a788621c0 x1874668972812288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827580.599825:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972812800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827580.599834:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972812288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827580.599845:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827580.599848:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827580.599850:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827580.599857:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827580.599858:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972812288:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827580.599863:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827580.599871:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4030 00000100:00080000:0.0:1787827580.599875:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827580.599879:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827580.599883:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972812416:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787827580.599884:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827580.599909:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972812928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827580.599910:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827580.599923:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827580.599926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827580/real 1787827580] req@ffff9c1a78860780 x1874668972812416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827580.599928:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827580.599931:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827580.599939:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4030 00000100:00080000:0.0:1787827580.599945:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827580.599950:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827580.599950:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972812416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827580.599958:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827580.599971:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972813056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827580.600001:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827580.600006:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827580.600009:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972812416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827580.600015:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972812544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827580.600031:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827580.600038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827580/real 1787827580] req@ffff9c1a78861e00 x1874668972812544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827580.600046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972812544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827580.600052:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827580.600055:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827580.600057:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972812544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827580.600061:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972812672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827580.600071:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827580.600077:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827580/real 1787827580] req@ffff9c1a78862580 x1874668972812672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827580.600085:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972812672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827580.600091:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827580.600092:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827580.600096:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972812672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827580.600101:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972812800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827580.600135:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827580.600146:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827580/real 1787827580] req@ffff9c1a78861680 x1874668972812800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827580.600159:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972812800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827580.600168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827580.600180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827580.600199:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972812800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827580.600213:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972812928:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827580.600231:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827580.600239:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827580/real 1787827580] req@ffff9c1a788630c0 x1874668972812928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827580.600247:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972812928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827580.600253:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827580.600255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827580.600257:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972812928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827580.600262:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972813056:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827580.600272:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827580.600279:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827580/real 1787827580] req@ffff9c1a78863c00 x1874668972813056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827580.600287:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972813056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827580.600292:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827580.600294:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827580.600296:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972813056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827581.111416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827581.111477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827581.367624:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827582.135479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827582.135479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827582.391585:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827583.159421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827583.159423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827583.415516:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827584.183477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827584.183477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827584.439502:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827585.207413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827585.207436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827585.463482:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827585.719494:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827585.719506:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827585.719511:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827585.719521:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4036 00000100:00080000:0.0:1787827585.719534:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827585.719542:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827585.719587:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972813184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827585.719618:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827585.719623:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827585.719626:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827585.719631:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4036 00000100:00080000:0.0:1787827585.719636:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827585.719680:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827585.719693:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827585.719716:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972813184:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827585.719718:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972813312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827585.719741:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827585.719745:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827585.719748:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827585.719753:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4036 00000100:00080000:0.0:1787827585.719775:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827585.719781:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827585.719788:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827585.719807:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972813440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827585.719816:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827585.719833:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827585.719837:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827585.719839:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827585.719846:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4036 00000100:00080000:0.0:1787827585.719850:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827585.719855:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827585.719862:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827585.719872:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827585/real 1787827585] req@ffff9c1a78860000 x1874668972813184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827585.719878:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972813568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827585.719914:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827585.719919:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827585.719923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972813184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827585.719924:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827585.719939:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4036 00000100:00080000:0.0:1787827585.719948:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827585.719957:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827585.719958:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827585.719962:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827585.719968:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827585.719976:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972813184:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827585.719985:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972813696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827585.720003:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827585.720025:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827585.720028:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827585.720032:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972813312:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827585.720039:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4036 00000100:00080000:0.0:1787827585.720047:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827585.720054:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827585.720062:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827585.720079:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827585.720084:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972813824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827585.720097:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827585.720101:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787827585.720120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827585/real 1787827585] req@ffff9c1a78863c00 x1874668972813312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827585.720122:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827585.720136:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4036 00000100:00080000:0.0:1787827585.720147:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827585.720158:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827585.720174:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827585.720180:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972813312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827585.720194:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827585.720218:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972813952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827585.720219:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827585.720227:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972813312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827585.720242:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972813440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827585.720272:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827585.720283:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827585/real 1787827585] req@ffff9c1a788630c0 x1874668972813440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827585.720294:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972813440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827585.720301:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827585.720304:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827585.720307:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972813440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827585.720312:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972813568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827585.720325:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827585.720333:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827585/real 1787827585] req@ffff9c1a78861680 x1874668972813568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827585.720343:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972813568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827585.720359:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827585.720364:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827585.720368:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972813568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827585.720377:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972813696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827585.720403:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827585.720459:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827585/real 1787827585] req@ffff9c1a78862580 x1874668972813696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827585.720478:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972813696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827585.720491:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827585.720495:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827585.720501:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972813696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827585.720523:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972813824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827585.720546:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827585.720557:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827585/real 1787827585] req@ffff9c1a78861e00 x1874668972813824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827585.720568:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972813824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827585.720575:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827585.720578:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827585.720582:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972813824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827585.720587:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972813952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827585.720600:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827585.720608:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827585/real 1787827585] req@ffff9c1a78860780 x1874668972813952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827585.720618:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972813952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827585.720625:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827585.720627:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827585.720630:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972813952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827586.231448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827586.231449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827586.487551:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827587.255408:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827587.255408:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827587.511585:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827588.279457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827588.279457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827588.535485:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827589.303420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827589.303435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827589.559462:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827590.327457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827590.327478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827590.583526:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827590.839533:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827590.839545:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827590.839551:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827590.839563:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4041 00000100:00080000:0.0:1787827590.839575:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827590.839583:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827590.839672:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972814080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827590.839703:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827590.839708:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827590.839710:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827590.839714:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4041 00000100:00080000:0.0:1787827590.839718:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827590.839723:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827590.839733:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827590.839749:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972814208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827590.839761:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827590.839765:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827590.839767:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827590.839770:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4041 00000100:00080000:0.0:1787827590.839774:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827590.839783:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827590.839783:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972814080:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787827590.839791:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827590.839803:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972814336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827590.839821:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827590.839824:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827590.839826:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827590.839829:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4041 00000100:00080000:0.0:1787827590.839832:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827590.839835:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827590.839839:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827590.839851:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972814464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827590.839880:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827590.839885:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827590.839887:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827590.839891:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4041 00000100:00080000:0.0:1787827590.839899:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827590.839907:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827590.839916:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827590.839921:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827590.839946:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972814592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827590.839968:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827590.839975:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827590.839979:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827590.839989:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827590/real 1787827590] req@ffff9c1a788621c0 x1874668972814080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827590.839990:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4041 00000100:00080000:0.0:1787827590.840005:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827590.840020:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827590.840030:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972814080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827590.840031:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827590.840055:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827590.840067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827590.840070:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972814720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827590.840074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972814080:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827590.840095:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827590.840107:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827590.840112:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827590.840116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972814208:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827590.840127:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4041 00000100:00080000:0.0:1787827590.840138:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827590.840161:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827590.840170:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827590.840194:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972814848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827590.840196:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827590.840238:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827590/real 1787827590] req@ffff9c1a78862940 x1874668972814208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827590.840266:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972814208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827590.840284:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827590.840289:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827590.840293:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972814208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827590.840342:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972814336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827590.840458:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827590.840479:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827590/real 1787827590] req@ffff9c1a78860b40 x1874668972814336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827590.840494:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972814336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827590.840505:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827590.840509:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827590.840513:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972814336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827590.840520:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972814464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827590.840546:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827590.840557:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827590/real 1787827590] req@ffff9c1a78861a40 x1874668972814464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827590.840579:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972814464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827590.840589:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827590.840592:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827590.840595:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972814464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827590.840601:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972814592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827590.840676:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827590.840689:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827590/real 1787827590] req@ffff9c1a78860f00 x1874668972814592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827590.840702:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972814592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827590.840710:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827590.840714:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827590.840717:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972814592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827590.840722:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972814720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827590.840740:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827590.840749:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827590/real 1787827590] req@ffff9c1a78862d00 x1874668972814720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827590.840760:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972814720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827590.840774:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827590.840777:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827590.840781:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972814720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827590.840786:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972814848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827590.840800:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827590.840809:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827590/real 1787827590] req@ffff9c1a788612c0 x1874668972814848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827590.840820:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972814848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827590.840828:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827590.840830:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827590.840858:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972814848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827591.351445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827591.351445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827591.607648:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827592.375441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827592.375441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827592.631508:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827593.399456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827593.399459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827593.655623:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827594.423465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827594.423465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827594.679565:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827595.447535:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827595.447535:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827595.703653:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827595.959487:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827595.959498:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827595.959503:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827595.959515:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4046 00000100:00080000:0.0:1787827595.959528:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827595.959537:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827595.959620:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972814976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827595.959651:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827595.959656:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827595.959658:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827595.959663:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4046 00000100:00080000:0.0:1787827595.959667:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827595.959673:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827595.959686:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827595.959706:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972815104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827595.959715:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827595.959721:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827595.959723:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827595.959725:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972814976:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827595.959729:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4046 00000100:00080000:0.0:1787827595.959737:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827595.959743:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827595.959747:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827595.959763:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972815232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827595.959770:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827595.959773:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827595.959782:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827595.959785:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4046 00000100:00080000:0.0:1787827595.959798:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827595.959802:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827595.959807:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827595.959813:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827595.959820:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972815360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827595.959831:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827595.959834:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827595.959836:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827595.959840:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4046 00000100:00080000:0.0:1787827595.959843:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827595.959845:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827595.959850:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827595.959868:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972815488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827595.959873:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827595/real 1787827595] req@ffff9c1a78863840 x1874668972814976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827595.959881:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827595.959892:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827595.959896:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827595.959910:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4046 00000100:00080000:0.0:1787827595.959916:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827595.959922:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827595.959928:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827595.959935:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972814976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827595.959938:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972815616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827595.960010:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827595.960012:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827595.960014:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827595.960022:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4046 00000100:00080000:0.0:1787827595.960028:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827595.960039:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827595.960048:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827595.960059:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827595.960060:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972815744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827595.960077:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827595.960086:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972814976:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827595.960114:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972815104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827595.960142:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827595.960159:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827595/real 1787827595] req@ffff9c1a788612c0 x1874668972815104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827595.960172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972815104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827595.960181:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827595.960187:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827595.960191:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972815104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827595.960200:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972815232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827595.960218:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827595.960227:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827595/real 1787827595] req@ffff9c1a78862d00 x1874668972815232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827595.960246:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972815232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827595.960255:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827595.960258:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827595.960262:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972815232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827595.960271:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972815360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827595.960299:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827595.960323:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827595/real 1787827595] req@ffff9c1a78860f00 x1874668972815360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827595.960335:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972815360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827595.960344:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827595.960402:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827595.960408:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972815360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827595.960418:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972815488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827595.960442:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827595.960456:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827595/real 1787827595] req@ffff9c1a78861a40 x1874668972815488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827595.960476:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972815488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827595.960483:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827595.960487:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827595.960489:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972815488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827595.960496:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972815616:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827595.960517:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827595.960535:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827595/real 1787827595] req@ffff9c1a78860b40 x1874668972815616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827595.960557:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972815616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827595.960565:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827595.960584:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827595.960587:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972815616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827595.960593:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972815744:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827595.960611:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827595.960620:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827595/real 1787827595] req@ffff9c1a78862940 x1874668972815744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827595.960632:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972815744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827595.960640:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827595.960643:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827595.960645:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972815744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827596.471493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827596.471494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827596.727533:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827597.495481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827597.495482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827597.751582:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827598.519439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827598.519444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827598.775590:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827599.543514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827599.543519:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827599.799706:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827600.567450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827600.567580:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827600.823531:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827601.079427:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827601.079438:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827601.079444:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827601.079456:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4051 00000100:00080000:0.0:1787827601.079470:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827601.079478:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827601.079526:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972815872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827601.079573:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827601.079576:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827601.079578:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827601.079582:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4051 00000100:00080000:0.0:1787827601.079585:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827601.079589:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827601.079601:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827601.079616:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972816000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827601.079625:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827601.079628:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827601.079630:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827601.079633:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4051 00000100:00080000:0.0:1787827601.079636:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827601.079639:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827601.079642:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827601.079653:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972816128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827601.079665:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827601.079673:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827601.079678:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827601.079680:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972815872:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827601.079691:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4051 00000100:00080000:0.0:1787827601.079702:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827601.079706:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827601.079710:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827601.079732:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972816256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827601.079745:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827601.079750:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827601.079754:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827601.079762:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4051 00000100:00080000:0.0:1787827601.079771:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827601.079778:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827601.079784:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827601.079788:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827601.079803:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972816384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827601.079816:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827601.079818:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827601.079820:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827601.079825:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4051 00000100:00080000:0.0:1787827601.079832:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827601.079835:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827601.079841:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827601.079853:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827601/real 1787827601] req@ffff9c1a788621c0 x1874668972815872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827601.079856:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972816512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827601.079952:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827601.079971:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827601.079978:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827601.079996:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4051 00000100:00080000:0.0:1787827601.080008:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827601.080017:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827601.080023:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827601.080033:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972815872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827601.080035:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972816640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827601.080055:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827601.080068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827601.080090:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972815872:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827601.080105:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972816000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827601.080156:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827601.080174:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827601/real 1787827601] req@ffff9c1a78860780 x1874668972816000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827601.080191:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972816000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827601.080202:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827601.080207:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827601.080211:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972816000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827601.080243:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972816128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827601.080279:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827601.080302:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827601/real 1787827601] req@ffff9c1a78861e00 x1874668972816128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827601.080327:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972816128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827601.080412:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827601.080417:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827601.080422:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972816128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827601.080431:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972816256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827601.080464:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827601.080479:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827601/real 1787827601] req@ffff9c1a78862580 x1874668972816256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827601.080494:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972816256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827601.080505:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827601.080509:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827601.080513:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972816256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827601.080520:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972816384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827601.080592:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827601.080607:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827601/real 1787827601] req@ffff9c1a78861680 x1874668972816384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827601.080640:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972816384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827601.080651:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827601.080655:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827601.080660:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972816384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827601.080668:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972816512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827601.080711:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827601.080723:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827601/real 1787827601] req@ffff9c1a788630c0 x1874668972816512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827601.080736:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972816512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827601.080745:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827601.080748:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827601.080752:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972816512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827601.080758:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972816640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827601.080776:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827601.080786:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827601/real 1787827601] req@ffff9c1a78863c00 x1874668972816640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827601.080799:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972816640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827601.080808:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827601.080811:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827601.080814:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972816640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827601.591459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827601.591479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827601.847521:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827602.615406:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827602.615409:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827602.871498:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827603.639485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827603.639486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827603.895534:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827604.663441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827604.663698:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827604.919432:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827605.687551:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827605.687551:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827605.943570:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827606.199431:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827606.199441:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827606.199445:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827606.199455:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4056 00000100:00080000:0.0:1787827606.199465:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827606.199471:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827606.199521:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972816768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827606.199548:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827606.199552:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827606.199553:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827606.199557:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4056 00000100:00080000:0.0:1787827606.199560:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827606.199564:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827606.199571:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827606.199584:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972816896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827606.199593:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827606.199596:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827606.199598:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827606.199601:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4056 00000100:00080000:0.0:1787827606.199605:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827606.199611:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827606.199614:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972816768:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787827606.199616:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827606.199630:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972817024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827606.199640:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827606.199643:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827606.199645:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827606.199648:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4056 00000100:00080000:0.0:1787827606.199651:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827606.199654:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827606.199658:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827606.199667:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972817152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827606.199674:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827606.199678:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827606.199682:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827606.199682:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827606.199689:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4056 00000100:00080000:0.0:1787827606.199694:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827606.199698:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827606.199702:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827606.199711:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972817280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827606.199724:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827606.199727:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827606.199729:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827606.199734:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4056 00000100:00100000:1.0:1787827606.199736:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827606/real 1787827606] req@ffff9c1a78860000 x1874668972816768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827606.199737:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827606.199743:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827606.199751:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827606.199763:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972817408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827606.199772:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827606.199775:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972816768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827606.199776:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827606.199780:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827606.199788:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4056 00000100:00080000:0.0:1787827606.199794:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827606.199801:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827606.199803:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827606.199804:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827606.199812:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827606.199818:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972817536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827606.199819:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972816768:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827606.199835:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972816896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827606.199863:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827606.199875:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827606/real 1787827606] req@ffff9c1a78863c00 x1874668972816896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827606.199908:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972816896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827606.199914:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827606.199916:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827606.199919:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972816896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827606.199937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972817024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827606.199952:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827606.199959:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827606/real 1787827606] req@ffff9c1a788630c0 x1874668972817024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827606.199967:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972817024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827606.199973:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827606.199975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827606.199977:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972817024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827606.199980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972817152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827606.199990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827606.200002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827606/real 1787827606] req@ffff9c1a78861680 x1874668972817152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827606.200010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972817152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827606.200015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827606.200017:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827606.200019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972817152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827606.200023:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972817280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827606.200034:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827606.200040:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827606/real 1787827606] req@ffff9c1a78862580 x1874668972817280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827606.200053:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972817280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827606.200058:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827606.200059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827606.200061:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972817280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827606.200064:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972817408:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827606.200075:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827606.200080:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827606/real 1787827606] req@ffff9c1a78861e00 x1874668972817408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827606.200087:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972817408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827606.200092:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827606.200094:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827606.200096:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972817408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827606.200099:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972817536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827606.200108:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827606.200113:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827606/real 1787827606] req@ffff9c1a78860780 x1874668972817536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827606.200120:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972817536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827606.200130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827606.200132:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827606.200134:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972817536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827606.711553:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827606.711553:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827606.967487:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827607.735434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827607.735434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827607.991520:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827608.759473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827608.759473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827609.015688:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827609.783491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827609.783493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827610.039567:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827610.807455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827610.807455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827611.063539:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827611.319532:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827611.319543:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827611.319564:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827611.319578:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4061 00000100:00080000:2.0:1787827611.319593:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827611.319600:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827611.319641:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972817664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827611.319674:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827611.319678:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827611.319697:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827611.319712:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4061 00000100:00080000:2.0:1787827611.319715:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827611.319719:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827611.319729:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827611.319743:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972817792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827611.319762:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827611.319766:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827611.319768:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827611.319775:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972817664:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827611.319776:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4061 00000100:00080000:2.0:1787827611.319799:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827611.319805:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827611.319809:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827611.319819:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972817920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827611.319828:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827611.319831:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827611.319833:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827611.319836:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4061 00000100:00080000:2.0:1787827611.319839:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827611.319842:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827611.319845:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827611.319855:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972818048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827611.319865:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827611.319867:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827611.319872:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827611.319874:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827611.319878:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4061 00000100:00080000:2.0:1787827611.319883:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827611.319886:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827611.319890:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827611.319899:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972818176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827611.319906:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827611.319909:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827611.319912:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827611/real 1787827611] req@ffff9c1a49d8de00 x1874668972817664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827611.319913:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827611.319922:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4061 00000100:00080000:2.0:1787827611.319928:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827611.319934:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827611.319939:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827611.319944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972817664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827611.319952:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972818304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827611.319971:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827611.319975:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827611.319980:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827611.319981:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827611.319985:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827611.319989:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972817664:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827611.319992:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4061 00000100:00080000:2.0:1787827611.319999:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827611.320003:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827611.320006:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827611.320014:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972818432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827611.320020:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972817792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827611.320054:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827611.320087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827611/real 1787827611] req@ffff9c1a49d8da40 x1874668972817792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827611.320106:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972817792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827611.320120:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827611.320126:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827611.320147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972817792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827611.320168:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972817920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827611.320201:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827611.320220:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827611/real 1787827611] req@ffff9c1a49d8cf00 x1874668972817920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827611.320234:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972817920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827611.320242:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827611.320246:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827611.320249:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972817920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827611.320255:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972818048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827611.320277:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827611.320287:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827611/real 1787827611] req@ffff9c1a49d8f0c0 x1874668972818048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827611.320298:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972818048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827611.320307:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827611.320310:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827611.320313:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972818048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827611.320319:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972818176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827611.320334:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827611.320355:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827611/real 1787827611] req@ffff9c1a49d8d680 x1874668972818176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827611.320366:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972818176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827611.320374:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827611.320377:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827611.320379:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972818176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827611.320385:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972818304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827611.320441:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827611.320453:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827611/real 1787827611] req@ffff9c1a49d8c780 x1874668972818304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827611.320504:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972818304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827611.320521:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827611.320524:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827611.320526:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972818304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827611.320544:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972818432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827611.320583:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827611.320594:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827611/real 1787827611] req@ffff9c1a49d8de00 x1874668972818432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827611.320614:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972818432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827611.320622:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827611.320625:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827611.320629:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972818432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827611.831449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827611.831484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827612.087520:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827612.855473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827612.855491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827613.111561:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827613.879431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827613.879465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827614.135559:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827614.903499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827614.903512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827615.159579:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827615.927585:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827615.927585:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827616.183545:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827616.439486:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827616.439495:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827616.439499:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827616.439508:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4066 00000100:00080000:2.0:1787827616.439519:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827616.439525:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827616.439563:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972818560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827616.439593:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827616.439596:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827616.439598:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827616.439602:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4066 00000100:00080000:2.0:1787827616.439604:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827616.439607:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827616.439616:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827616.439626:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972818688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827616.439636:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827616.439639:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827616.439641:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827616.439645:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4066 00000100:00080000:2.0:1787827616.439648:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827616.439651:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827616.439655:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827616.439663:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972818560:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827616.439665:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972818816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827616.439683:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827616.439686:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827616.439687:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827616.439690:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4066 00000100:00080000:2.0:1787827616.439692:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827616.439695:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827616.439699:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827616.439710:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972818944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827616.439718:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827616.439721:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827616.439722:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827616.439725:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4066 00000100:00080000:2.0:1787827616.439737:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827616.439740:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827616.439746:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827616.439748:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827616.439759:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972819072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827616.439783:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827616.439785:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827616.439786:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827616.439790:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4066 00000100:00080000:2.0:1787827616.439793:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827616.439796:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827616.439800:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827616.439809:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972819200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827616.439816:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827616.439821:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827616/real 1787827616] req@ffff9c1a49d8c000 x1874668972818560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827616.439821:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827616.439825:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827616.439850:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4066 00000100:00080000:2.0:1787827616.439857:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827616.439862:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827616.439867:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827616.439872:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972818560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827616.439884:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972819328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827616.439886:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827616.439897:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827616.439907:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972818560:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827616.439916:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972818688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827616.439942:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827616.439953:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827616/real 1787827616] req@ffff9c1a49d8e1c0 x1874668972818688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827616.439970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972818688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827616.439977:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827616.439980:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827616.439983:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972818688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827616.440006:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972818816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827616.440022:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827616.440046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827616/real 1787827616] req@ffff9c1a49d8fc00 x1874668972818816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827616.440055:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972818816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827616.440060:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827616.440062:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827616.440064:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972818816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827616.440068:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972818944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827616.440081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827616.440109:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827616/real 1787827616] req@ffff9c1a49d8e580 x1874668972818944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827616.440125:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972818944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827616.440146:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827616.440150:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827616.440156:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972818944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827616.440164:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972819072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827616.440184:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827616.440195:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827616/real 1787827616] req@ffff9c1a49d8c3c0 x1874668972819072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827616.440205:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972819072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827616.440212:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827616.440215:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827616.440217:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972819072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827616.440222:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972819200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827616.440235:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827616.440242:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827616/real 1787827616] req@ffff9c1a49d8ed00 x1874668972819200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827616.440252:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972819200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827616.440258:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827616.440261:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827616.440264:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972819200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827616.440269:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972819328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827616.440281:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827616.440288:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827616/real 1787827616] req@ffff9c1a49d8cb40 x1874668972819328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827616.440297:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972819328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827616.440318:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827616.440320:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827616.440322:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972819328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827616.951544:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827616.951545:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827617.207593:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827617.975434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827617.975441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827618.231490:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827618.999467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827618.999467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827619.255533:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827620.023435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827620.023448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827620.279512:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827621.047498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827621.047499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827621.303504:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827621.559420:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827621.559428:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827621.559432:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827621.559440:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4071 00000100:00080000:2.0:1787827621.559449:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827621.559454:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827621.559485:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972819456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827621.559506:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827621.559509:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827621.559511:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827621.559513:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4071 00000100:00080000:2.0:1787827621.559515:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827621.559518:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827621.559525:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827621.559533:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972819584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827621.559542:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827621.559544:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827621.559546:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827621.559548:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4071 00000100:00080000:2.0:1787827621.559551:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827621.559553:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827621.559557:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827621.559566:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972819712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827621.559575:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827621.559582:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827621.559587:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827621.559588:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972819456:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827621.559593:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4071 00000100:00080000:2.0:1787827621.559599:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827621.559608:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827621.559611:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827621.559617:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972819840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827621.559622:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827621.559625:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827621.559626:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827621.559628:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4071 00000100:00080000:2.0:1787827621.559632:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827621.559635:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827621.559638:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827621.559644:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972819968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827621.559655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827621.559658:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827621.559659:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827621.559662:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4071 00000100:00080000:2.0:1787827621.559665:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827621.559668:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827621.559670:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827621.559676:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972820096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827621.559683:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827621.559686:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827621.559687:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827621.559689:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4071 00000100:00080000:2.0:1787827621.559691:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827621.559693:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827621.559696:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827621.559704:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827621.559710:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972820224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827621.559767:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827621/real 1787827621] req@ffff9c1a49d8cb40 x1874668972819456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.559791:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972819456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.559805:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.559810:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827621.559816:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972819456:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827621.559826:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972819584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827621.559858:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827621.559873:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827621/real 1787827621] req@ffff9c1a49d8ed00 x1874668972819584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.559889:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972819584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.559899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.559903:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827621.559907:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972819584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827621.559943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972819712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827621.559968:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827621.559981:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827621/real 1787827621] req@ffff9c1a49d8c3c0 x1874668972819712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.559994:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972819712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.560004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.560008:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827621.560011:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972819712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827621.560019:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972819840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827621.560040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827621.560053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827621/real 1787827621] req@ffff9c1a49d8e580 x1874668972819840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.560074:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972819840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.560084:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.560088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827621.560092:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972819840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827621.560099:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972819968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827621.560121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827621.560134:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827621/real 1787827621] req@ffff9c1a49d8fc00 x1874668972819968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.560146:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972819968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.560154:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.560158:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827621.560161:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972819968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827621.560167:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972820096:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827621.560188:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827621.560200:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827621/real 1787827621] req@ffff9c1a49d8e1c0 x1874668972820096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.560212:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972820096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.560227:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.560231:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827621.560234:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972820096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827621.560241:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972820224:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827621.560263:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827621.560274:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827621/real 1787827621] req@ffff9c1a49d8c000 x1874668972820224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827621.560287:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972820224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827621.560295:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827621.560298:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827621.560301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972820224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827622.071444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827622.071452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827622.327573:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827623.095482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827623.095482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827623.351552:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827624.119519:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827624.119529:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827624.375528:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827625.143558:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827625.143744:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827625.399577:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827626.167511:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827626.167511:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827626.423638:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827626.679536:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827626.679549:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827626.679556:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827626.679568:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4077 00000100:00080000:0.0:1787827626.679579:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827626.679586:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827626.679627:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972820352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827626.679659:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827626.679663:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827626.679665:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827626.679669:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4077 00000100:00080000:0.0:1787827626.679672:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827626.679676:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827626.679687:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827626.679716:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972820480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827626.679728:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827626.679732:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827626.679736:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827626.679741:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4077 00000100:00080000:0.0:1787827626.679746:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827626.679751:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827626.679759:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827626.679771:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972820352:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827626.679790:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972820608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827626.679817:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827626.679824:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827626.679829:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827626.679837:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4077 00000100:00080000:0.0:1787827626.679846:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827626.679854:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827626.679864:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827626.679886:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972820736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827626.679901:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827626.679903:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827626.679916:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827626.679919:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827626.679927:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4077 00000100:00080000:0.0:1787827626.679933:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827626.679937:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827626.679942:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827626.679963:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972820864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827626.679974:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827626.679978:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827626.679981:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827626.679987:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4077 00000100:00080000:0.0:1787827626.679995:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827626.680001:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827626.680013:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827626/real 1787827626] req@ffff9c1a788621c0 x1874668972820352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827626.680015:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827626.680046:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972820992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827626.680060:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827626.680064:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972820352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827626.680066:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827626.680075:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827626.680100:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4077 00000100:00080000:0.0:1787827626.680114:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827626.680115:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827626.680124:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827626.680126:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787827626.680136:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827626.680145:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972820352:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827626.680159:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972821120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827626.680170:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972820480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827626.680226:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827626.680243:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827626/real 1787827626] req@ffff9c1a78862940 x1874668972820480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827626.680258:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972820480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827626.680269:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827626.680273:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827626.680277:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972820480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827626.680312:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972820608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827626.680334:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827626.680441:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827626/real 1787827626] req@ffff9c1a78860b40 x1874668972820608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827626.680460:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972820608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827626.680471:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827626.680475:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827626.680479:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972820608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827626.680487:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972820736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827626.680521:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827626.680543:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827626/real 1787827626] req@ffff9c1a78861a40 x1874668972820736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827626.680566:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972820736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827626.680583:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827626.680589:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827626.680596:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972820736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827626.680607:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972820864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827626.680637:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827626.680650:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827626/real 1787827626] req@ffff9c1a78860f00 x1874668972820864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827626.680664:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972820864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827626.680674:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827626.680678:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827626.680681:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972820864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827626.680689:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972820992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827626.680709:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827626.680719:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827626/real 1787827626] req@ffff9c1a78862d00 x1874668972820992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827626.680732:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972820992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827626.680742:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827626.680746:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827626.680749:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972820992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827626.680756:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972821120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827626.680773:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827626.680782:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827626/real 1787827626] req@ffff9c1a788612c0 x1874668972821120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827626.680794:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972821120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827626.680815:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827626.680818:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827626.680821:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972821120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827627.191565:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827627.191570:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827627.447596:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827628.215453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827628.215454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827628.471528:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827629.239472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827629.239483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827629.495639:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827630.263446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827630.263458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827630.519583:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827631.287466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827631.287591:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827631.543537:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827631.799513:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827631.799525:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827631.799531:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827631.799544:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4082 00000100:00080000:0.0:1787827631.799557:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827631.799584:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827631.799621:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972821248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827631.799673:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827631.799679:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827631.799681:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827631.799687:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4082 00000100:00080000:0.0:1787827631.799691:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827631.799696:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827631.799708:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827631.799737:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972821376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827631.799766:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827631.799769:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827631.799771:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827631.799775:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4082 00000100:00080000:0.0:1787827631.799780:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827631.799783:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972821248:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827631.799786:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827631.799795:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827631.799810:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972821504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827631.799827:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827631.799832:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827631.799835:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827631.799841:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4082 00000100:00080000:0.0:1787827631.799850:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827631.799857:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827631.799865:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827631.799870:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827631.799888:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972821632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827631.799905:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827631.799910:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827631.799914:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827631.799920:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4082 00000100:00100000:1.0:1787827631.799926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827631/real 1787827631] req@ffff9c1a78863840 x1874668972821248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827631.799930:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827631.799942:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827631.799956:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827631.799970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972821248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827631.799974:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972821760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827631.800024:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827631.800053:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827631.800058:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827631.800067:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4082 00000100:00080000:1.0:1787827631.800069:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827631.800073:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827631.800080:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827631.800086:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827631.800095:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972821248:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787827631.800097:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827631.800117:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972821376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827631.800121:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972821888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827631.800166:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827631.800171:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827631.800172:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827631.800176:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4082 00000100:00080000:0.0:1787827631.800180:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827631.800185:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827631.800191:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827631.800223:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827631.800225:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972822016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827631.800267:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827631/real 1787827631] req@ffff9c1a788612c0 x1874668972821376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827631.800301:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972821376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827631.800309:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827631.800312:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827631.800315:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972821376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827631.800396:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972821504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827631.800456:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827631.800475:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827631/real 1787827631] req@ffff9c1a78862d00 x1874668972821504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827631.800492:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972821504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827631.800502:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827631.800507:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827631.800512:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972821504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827631.800526:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972821632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827631.800569:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827631.800582:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827631/real 1787827631] req@ffff9c1a78860f00 x1874668972821632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827631.800591:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972821632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827631.800599:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827631.800601:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827631.800604:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972821632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827631.800609:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972821760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827631.800622:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827631.800629:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827631/real 1787827631] req@ffff9c1a78861a40 x1874668972821760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827631.800659:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972821760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827631.800668:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827631.800671:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827631.800674:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972821760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827631.800680:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972821888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827631.800699:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827631.800713:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827631/real 1787827631] req@ffff9c1a78860b40 x1874668972821888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827631.800748:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972821888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827631.800760:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827631.800764:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827631.800768:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972821888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827631.800776:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972822016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827631.800801:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827631.800815:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827631/real 1787827631] req@ffff9c1a78862940 x1874668972822016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827631.800831:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972822016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827631.800838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827631.800841:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827631.800845:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972822016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827632.311494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827632.311497:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827632.567502:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827633.335581:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827633.335581:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827633.591470:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827634.359471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827634.359507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827634.615589:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827635.383437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827635.383439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827635.639490:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827636.407453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827636.407453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827636.663514:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827636.919450:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827636.919459:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827636.919462:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827636.919470:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4087 00000100:00080000:2.0:1787827636.919479:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827636.919485:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827636.919516:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972822144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827636.919538:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827636.919541:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827636.919542:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827636.919545:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4087 00000100:00080000:2.0:1787827636.919547:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827636.919550:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827636.919557:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827636.919565:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972822272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827636.919572:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827636.919574:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827636.919575:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827636.919577:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4087 00000100:00080000:2.0:1787827636.919579:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827636.919581:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827636.919584:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827636.919589:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972822400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827636.919596:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827636.919599:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827636.919601:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827636.919603:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4087 00000100:00080000:2.0:1787827636.919605:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827636.919608:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827636.919611:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827636.919620:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972822144:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827636.919620:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972822528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827636.919640:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827636.919643:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827636.919644:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827636.919647:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4087 00000100:00080000:2.0:1787827636.919649:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827636.919652:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827636.919655:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827636.919664:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972822656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827636.919671:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827636.919673:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827636.919675:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827636.919677:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4087 00000100:00080000:2.0:1787827636.919680:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827636.919682:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827636.919685:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827636.919692:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972822784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827636.919698:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827636.919701:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827636.919702:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827636.919706:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4087 00000100:00080000:2.0:1787827636.919709:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827636.919712:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827636.919715:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972822272:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787827636.919718:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827636.919735:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972822912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827636.919747:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972822400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827636.919772:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972822528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827636.919786:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972822656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827636.919800:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972822784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827636.919814:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972822912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827637.431445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827637.431483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827637.687561:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827638.455438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827638.455438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827638.711538:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827639.479409:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827639.479419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827639.735522:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827639.991644:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787827639.991689:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827639.991705:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827639.991743:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827639.991752:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827639.991760:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827639.991771:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827639.991781:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787827639.991792:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827636/real 1787827639] req@ffff9c1a49d8f0c0 x1874668972822528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827639.991793:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787827639.991825:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972822528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.991837:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.991842:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827639.991847:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972822528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827639.991854:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827636/real 1787827639] req@ffff9c1a49d8cf00 x1874668972822656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.991865:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972822656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.991873:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.991877:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827639.991880:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972822656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827639.991887:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827636/real 1787827639] req@ffff9c1a49d8da40 x1874668972822784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.991904:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972822784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.991911:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.991914:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827639.991917:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972822784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827639.991924:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827636/real 1787827639] req@ffff9c1a49d8c000 x1874668972822912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.991936:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972822912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.991943:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.991947:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827639.991950:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972822912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827639.991957:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827636/real 1787827639] req@ffff9c1a49d8de00 x1874668972822144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.991969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972822144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.991977:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.991981:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827639.991985:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972822144:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827639.991992:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827636/real 1787827639] req@ffff9c1a49d8c780 x1874668972822272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.992005:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972822272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.992014:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.992016:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827639.992020:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972822272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827639.992026:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827636/real 1787827639] req@ffff9c1a49d8d680 x1874668972822400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827639.992037:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972822400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827639.992044:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827639.992047:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827639.992050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972822400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827640.503454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827640.503454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827640.759491:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827641.527413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827641.527414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827641.783478:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827642.039426:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827642.039435:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827642.039438:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827642.039448:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4092 00000100:00080000:2.0:1787827642.039457:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827642.039463:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827642.039493:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972823040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827642.039517:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827642.039520:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827642.039521:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827642.039524:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4092 00000100:00080000:2.0:1787827642.039526:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827642.039529:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827642.039536:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827642.039545:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972823168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827642.039553:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827642.039556:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827642.039558:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827642.039561:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4092 00000100:00080000:2.0:1787827642.039564:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827642.039566:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827642.039569:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827642.039575:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972823296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827642.039585:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827642.039590:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787827642.039598:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972823040:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827642.039599:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827642.039612:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4092 00000100:00080000:2.0:1787827642.039620:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827642.039623:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827642.039626:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827642.039633:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972823424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827642.039640:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827642.039643:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827642.039644:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827642.039649:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4092 00000100:00080000:2.0:1787827642.039653:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827642.039658:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827642.039664:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827642.039678:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972823552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827642.039689:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827642.039693:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827642.039695:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827642.039699:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4092 00000100:00080000:2.0:1787827642.039705:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827642.039709:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787827642.039717:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787827642.039717:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827642.039738:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972823680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827642.039753:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827642.039757:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827642.039758:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827642.039763:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4092 00000100:00080000:2.0:1787827642.039769:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827642.039775:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827642.039782:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827642.039796:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827642/real 1787827642] req@ffff9c1a49d8e1c0 x1874668972823040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827642.039811:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972823808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827642.039832:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972823040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827642.039851:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827642.039858:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827642.039865:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972823040:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827642.039893:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972823168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827642.039928:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827642.039944:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827642/real 1787827642] req@ffff9c1a49d8fc00 x1874668972823168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827642.039957:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972823168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827642.039966:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827642.039969:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827642.039973:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972823168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827642.040009:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972823296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827642.040034:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827642.040048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827642/real 1787827642] req@ffff9c1a49d8e580 x1874668972823296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827642.040061:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972823296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827642.040071:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827642.040075:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827642.040078:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972823296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827642.040086:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972823424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827642.040112:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827642.040127:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827642/real 1787827642] req@ffff9c1a49d8c3c0 x1874668972823424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827642.040141:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972823424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827642.040161:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827642.040165:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827642.040168:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972823424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827642.040176:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972823552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827642.040200:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827642.040214:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827642/real 1787827642] req@ffff9c1a49d8ed00 x1874668972823552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827642.040229:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972823552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827642.040240:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827642.040244:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827642.040263:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972823552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827642.040272:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972823680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827642.040308:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827642.040321:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827642/real 1787827642] req@ffff9c1a49d8cb40 x1874668972823680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827642.040333:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972823680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827642.040343:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827642.040403:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827642.040408:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972823680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827642.040416:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972823808:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827642.040447:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827642.040460:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827642/real 1787827642] req@ffff9c1a49d8d680 x1874668972823808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827642.040474:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972823808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827642.040483:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827642.040487:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827642.040491:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972823808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827642.551523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827642.551524:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827642.807546:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827643.575395:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827643.575409:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827643.831491:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827644.599443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827644.599443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827644.855553:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827645.623478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827645.623478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827645.879468:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827646.647450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827646.647450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827646.903535:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827647.159436:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827647.159444:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827647.159448:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827647.159456:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4097 00000100:00080000:2.0:1787827647.159465:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827647.159470:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827647.159501:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972823936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827647.159523:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827647.159526:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827647.159528:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827647.159531:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4097 00000100:00080000:2.0:1787827647.159533:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827647.159535:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827647.159544:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827647.159552:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972824064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827647.159560:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827647.159562:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827647.159564:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827647.159567:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972823936:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827647.159569:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4097 00000100:00080000:2.0:1787827647.159574:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827647.159578:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827647.159581:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827647.159590:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972824192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827647.159597:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827647.159600:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827647.159601:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827647.159604:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4097 00000100:00080000:2.0:1787827647.159607:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827647.159610:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827647.159613:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827647.159619:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972824320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827647.159625:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827647.159628:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827647.159629:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827647.159634:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4097 00000800:00000100:1.0:1787827647.159638:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827647.159639:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827647.159643:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827647.159647:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827647.159655:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972824448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827647.159661:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827647.159663:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827647.159665:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827647.159667:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4097 00000100:00080000:2.0:1787827647.159670:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827647.159673:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827647.159676:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827647.159683:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827647/real 1787827647] req@ffff9c1a49d8c780 x1874668972823936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827647.159686:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972824576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827647.159712:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827647.159714:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827647.159716:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827647.159721:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4097 00000100:00100000:1.0:1787827647.159722:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972823936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827647.159724:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827647.159730:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827647.159737:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827647.159742:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827647.159748:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972824704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827647.159749:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827647.159756:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972823936:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827647.159785:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972824064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827647.159804:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827647.159813:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827647/real 1787827647] req@ffff9c1a49d8de00 x1874668972824064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.159822:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972824064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.159828:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.159830:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827647.159832:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972824064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827647.159835:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972824192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827647.159846:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827647.159852:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827647/real 1787827647] req@ffff9c1a49d8c000 x1874668972824192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.159859:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972824192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.159864:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.159866:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827647.159867:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972824192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827647.159871:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972824320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827647.159881:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827647.159887:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827647/real 1787827647] req@ffff9c1a49d8da40 x1874668972824320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.159894:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972824320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.159900:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.159901:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827647.159903:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972824320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827647.159907:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972824448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827647.159916:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827647.159922:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827647/real 1787827647] req@ffff9c1a49d8cf00 x1874668972824448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.159929:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972824448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.159934:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.159935:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827647.159937:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972824448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827647.159940:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972824576:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827647.159951:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827647.159956:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827647/real 1787827647] req@ffff9c1a49d8f0c0 x1874668972824576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.159969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972824576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.159974:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.159976:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827647.159977:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972824576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827647.159981:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972824704:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827647.159990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827647.159995:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827647/real 1787827647] req@ffff9c1a49d8d680 x1874668972824704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827647.160003:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972824704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827647.160008:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827647.160009:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827647.160011:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972824704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827647.671453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827647.671453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827647.927440:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827648.695442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827648.695443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827648.951478:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827649.719430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827649.719434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827649.975508:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827650.743452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827650.743452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827650.999492:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827651.767460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827651.767460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827652.023503:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827652.279421:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827652.279429:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827652.279432:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827652.279440:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4102 00000100:00080000:1.0:1787827652.279448:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827652.279454:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827652.279484:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972824832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827652.279505:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827652.279508:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827652.279509:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827652.279512:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4102 00000100:00080000:1.0:1787827652.279514:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827652.279516:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827652.279524:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827652.279533:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972824960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827652.279540:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827652.279543:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827652.279544:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827652.279550:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972824832:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827652.279550:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4102 00000100:00080000:1.0:1787827652.279555:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827652.279559:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827652.279562:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827652.279570:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972825088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827652.279577:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827652.279579:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827652.279582:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827652.279584:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4102 00000100:00080000:1.0:1787827652.279598:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827652.279601:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827652.279604:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827652.279610:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827652.279614:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972825216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827652.279621:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827652.279624:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827652.279625:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827652.279628:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4102 00000100:00080000:1.0:1787827652.279631:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827652.279633:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827652.279636:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827652.279644:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972825344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827652.279649:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827652/real 1787827652] req@ffff9c1a78a7a940 x1874668972824832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827652.279653:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827652.279661:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827652.279664:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827652.279670:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4102 00000100:00080000:1.0:1787827652.279673:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827652.279676:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972824832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827652.279679:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827652.279685:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827652.279692:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827652.279696:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972825472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827652.279698:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827652.279705:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972824832:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827652.279712:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827652.279715:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827652.279716:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827652.279720:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4102 00000100:00080000:1.0:1787827652.279722:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827652.279724:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827652.279727:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827652.279730:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972824960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827652.279736:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972825600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827652.279751:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827652.279763:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827652/real 1787827652] req@ffff9c1a78a783c0 x1874668972824960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827652.279776:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972824960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827652.279782:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827652.279784:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827652.279786:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972824960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827652.279790:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972825088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827652.279800:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827652.279806:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827652/real 1787827652] req@ffff9c1a78a7a1c0 x1874668972825088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827652.279813:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972825088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827652.279818:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827652.279820:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827652.279822:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972825088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827652.279825:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972825216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827652.279835:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827652.279841:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827652/real 1787827652] req@ffff9c1a78a78780 x1874668972825216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827652.279848:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972825216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827652.279857:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827652.279859:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827652.279861:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972825216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827652.279864:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972825344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827652.279874:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827652.279881:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827652/real 1787827652] req@ffff9c1a78a7bc00 x1874668972825344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827652.279892:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972825344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827652.279901:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827652.279904:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827652.279907:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972825344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827652.279914:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972825472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827652.279935:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827652.279946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827652/real 1787827652] req@ffff9c1a78a7b0c0 x1874668972825472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827652.279959:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972825472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827652.279967:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827652.279971:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827652.279975:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972825472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827652.279993:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972825600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827652.280014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827652.280023:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827652/real 1787827652] req@ffff9c1a78a79e00 x1874668972825600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827652.280030:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972825600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827652.280036:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827652.280038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827652.280040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972825600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827652.791429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827652.791429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827653.047478:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827653.815451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827653.815451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827654.071468:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827654.839431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827654.839464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827655.095485:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827655.863473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827655.863479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827656.119456:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827656.887412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827656.887418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827657.143526:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827657.399499:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827657.399507:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827657.399511:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827657.399519:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4107 00000100:00080000:2.0:1787827657.399528:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827657.399534:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827657.399566:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972825728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827657.399592:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827657.399595:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827657.399597:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827657.399599:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4107 00000100:00080000:2.0:1787827657.399602:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827657.399604:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827657.399612:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827657.399620:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972825856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827657.399628:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827657.399630:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827657.399632:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827657.399634:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4107 00000100:00080000:2.0:1787827657.399638:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827657.399642:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827657.399647:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827657.399655:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972825728:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827657.399661:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972825984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827657.399678:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827657.399682:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827657.399684:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827657.399698:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4107 00000100:00080000:2.0:1787827657.399703:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827657.399706:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827657.399710:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827657.399725:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972826112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827657.399731:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827657.399745:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827657.399749:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827657.399752:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827657.399757:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4107 00000100:00080000:2.0:1787827657.399762:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827657.399766:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827657.399771:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827657.399778:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827657/real 1787827657] req@ffff9c1a49d8cb40 x1874668972825728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827657.399783:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972826240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827657.399798:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827657.399805:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827657.399808:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827657.399812:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4107 00000100:00080000:2.0:1787827657.399816:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827657.399819:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827657.399824:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972825728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827657.399825:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827657.399838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827657.399838:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972826368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827657.399848:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827657.399851:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827657.399855:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787827657.399857:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972825728:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827657.399858:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827657.399870:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972825856:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827657.399875:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4107 00000100:00080000:2.0:1787827657.399881:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827657.399885:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827657.399891:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827657.399904:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827657.399909:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972826496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827657.399925:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827657/real 1787827657] req@ffff9c1a49d8ed00 x1874668972825856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827657.399941:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972825856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827657.399951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827657.399955:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827657.399959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972825856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827657.399995:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972825984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827657.400013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827657.400023:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827657/real 1787827657] req@ffff9c1a49d8c3c0 x1874668972825984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827657.400036:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972825984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827657.400045:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827657.400048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827657.400051:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972825984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827657.400058:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972826112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827657.400080:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827657.400092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827657/real 1787827657] req@ffff9c1a49d8e580 x1874668972826112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827657.400114:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972826112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827657.400124:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827657.400128:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827657.400132:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972826112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827657.400136:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972826240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827657.400170:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827657.400180:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827657/real 1787827657] req@ffff9c1a49d8fc00 x1874668972826240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827657.400188:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972826240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827657.400196:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827657.400199:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827657.400202:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972826240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827657.400208:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972826368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827657.400229:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827657.400241:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827657/real 1787827657] req@ffff9c1a49d8e1c0 x1874668972826368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827657.400262:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972826368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827657.400271:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827657.400274:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827657.400278:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972826368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827657.400284:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972826496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827657.400301:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827657.400309:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827657/real 1787827657] req@ffff9c1a49d8d680 x1874668972826496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827657.400317:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972826496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827657.400323:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827657.400325:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827657.400327:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972826496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827657.911447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827657.911447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827658.167545:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827658.935437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827658.935717:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827659.191466:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827659.959415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827659.959432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827660.215574:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827660.983414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827660.983443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827661.239497:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827662.007443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827662.007444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827662.263482:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827662.519453:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827662.519460:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827662.519463:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827662.519472:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4112 00000100:00080000:2.0:1787827662.519482:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827662.519496:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827662.519535:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972826624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827662.519568:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827662.519572:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827662.519574:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827662.519578:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4112 00000100:00080000:2.0:1787827662.519583:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827662.519587:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827662.519597:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827662.519611:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972826752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827662.519616:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972826624:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827662.519634:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827662.519638:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827662.519640:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827662.519645:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4112 00000100:00080000:2.0:1787827662.519650:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827662.519655:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827662.519661:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827662.519675:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972826880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827662.519676:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827662.519691:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827662.519696:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827662.519699:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827662.519706:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4112 00000100:00080000:2.0:1787827662.519710:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827662.519715:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827662.519721:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827662.519727:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827662/real 1787827662] req@ffff9c1a49d8f0c0 x1874668972826624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827662.519736:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972827008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827662.519750:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827662.519755:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972826624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827662.519756:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827662.519761:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827662.519770:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4112 00000100:00080000:0.0:1787827662.519775:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827662.519776:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827662.519783:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827662.519785:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827662.519794:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972826624:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787827662.519794:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827662.519810:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972827136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827662.519823:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827662.519827:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972826752:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827662.519828:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827662.519833:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827662.519840:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4112 00000100:00080000:2.0:1787827662.519845:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827662.519850:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787827662.519855:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787827662.519857:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827662.519868:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827662/real 1787827662] req@ffff9c1a49d8cf00 x1874668972826752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827662.519874:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972827264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827662.519883:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972826752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827662.519895:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827662.519900:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827662.519901:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827662.519906:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827662.519910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827662.519916:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4112 00000100:00100000:0.0:1787827662.519920:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972826752:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827662.519923:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827662.519932:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972826880:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827662.519933:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827662.519942:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827662.519958:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972827392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827662.519968:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827662.519988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827662/real 1787827662] req@ffff9c1a49d8da40 x1874668972826880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827662.520002:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972826880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827662.520008:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827662.520011:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827662.520013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972826880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827662.520018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972827008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827662.520034:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827662.520041:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827662/real 1787827662] req@ffff9c1a49d8c000 x1874668972827008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827662.520049:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972827008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827662.520054:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827662.520056:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827662.520068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972827008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827662.520083:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972827136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827662.520095:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827662.520102:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827662/real 1787827662] req@ffff9c1a49d8de00 x1874668972827136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827662.520128:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972827136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827662.520134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827662.520136:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827662.520138:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972827136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827662.520142:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972827264:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827662.520161:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827662.520173:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827662/real 1787827662] req@ffff9c1a49d8c780 x1874668972827264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827662.520186:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972827264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827662.520195:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827662.520198:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827662.520202:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972827264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827662.520208:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972827392:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827662.520227:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827662.520239:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827662/real 1787827662] req@ffff9c1a49d8f0c0 x1874668972827392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827662.520252:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972827392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827662.520260:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827662.520263:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827662.520265:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972827392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827663.031444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827663.031453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827663.287475:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827664.055433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827664.055436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827664.311515:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827665.079417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827665.079454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827665.335504:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827666.103476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827666.103493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827666.359581:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827667.127461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827667.127461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827667.383486:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827667.639431:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827667.639442:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827667.639447:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827667.639458:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4117 00000100:00080000:2.0:1787827667.639480:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827667.639489:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827667.639525:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972827520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827667.639553:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827667.639558:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827667.639561:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827667.639567:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4117 00000100:00080000:2.0:1787827667.639573:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827667.639578:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827667.639586:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972827520:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787827667.639589:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827667.639610:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972827648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827667.639625:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827667.639629:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827667.639631:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827667.639636:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4117 00000100:00080000:2.0:1787827667.639642:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827667.639647:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827667.639652:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827667.639658:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827667.639667:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972827776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827667.639680:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827667.639685:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827667.639688:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827667.639692:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827667/real 1787827667] req@ffff9c1a49d8d680 x1874668972827520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827667.639696:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4117 00000100:00080000:2.0:1787827667.639704:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827667.639711:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827667.639716:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972827520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827667.639718:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787827667.639735:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827667.639740:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972827904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827667.639742:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827667.639755:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827667.639756:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972827520:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827667.639761:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827667.639765:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827667.639771:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4117 00000100:00080000:2.0:1787827667.639775:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827667.639779:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827667.639785:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972827648:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787827667.639785:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827667.639806:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972828032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827667.639818:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787827667.639823:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827667.639825:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827667.639829:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827667.639835:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4117 00000100:00080000:2.0:1787827667.639840:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827667.639846:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827667/real 1787827667] req@ffff9c1a49d8e1c0 x1874668972827648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827667.639846:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827667.639866:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827667.639867:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972827648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827667.639880:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827667.639882:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972828160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827667.639886:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827667.639894:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972827648:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827667.639906:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827667.639909:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972827776:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827667.639913:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827667.639918:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827667.639923:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4117 00000100:00080000:2.0:1787827667.639927:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827667.639931:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827667.639936:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827667.639944:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827667.639950:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972828288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827667.639961:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827667/real 1787827667] req@ffff9c1a49d8fc00 x1874668972827776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827667.639974:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972827776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827667.639981:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827667.639985:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827667.639988:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972827776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827667.639994:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972827904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827667.640012:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827667.640023:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827667/real 1787827667] req@ffff9c1a49d8e580 x1874668972827904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827667.640032:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972827904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827667.640039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827667.640042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827667.640044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972827904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827667.640066:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972828032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827667.640103:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827667.640112:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827667/real 1787827667] req@ffff9c1a49d8c3c0 x1874668972828032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827667.640120:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972828032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827667.640126:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827667.640128:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827667.640130:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972828032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827667.640133:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972828160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827667.640144:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827667.640150:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827667/real 1787827667] req@ffff9c1a49d8ed00 x1874668972828160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827667.640158:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972828160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827667.640163:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827667.640165:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827667.640167:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972828160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827667.640170:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972828288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827667.640180:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827667.640185:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827667/real 1787827667] req@ffff9c1a49d8cb40 x1874668972828288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827667.640193:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972828288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827667.640198:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827667.640199:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827667.640201:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972828288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827668.151453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827668.151453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827668.407527:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827669.175454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827669.175454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827669.431437:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827670.199456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827670.199459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827670.455533:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827671.223447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827671.223447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827671.479513:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827672.247437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827672.247437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827672.503524:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827672.759466:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827672.759476:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827672.759480:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827672.759490:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4123 00000100:00080000:0.0:1787827672.759499:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827672.759507:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827672.759542:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972828416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827672.759571:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827672.759575:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827672.759577:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827672.759582:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4123 00000100:00080000:0.0:1787827672.759585:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827672.759589:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827672.759599:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827672.759613:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972828544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827672.759616:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972828416:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827672.759632:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827672.759636:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827672.759639:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827672.759643:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4123 00000100:00080000:0.0:1787827672.759648:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827672.759651:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827672.759657:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827672.759669:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972828672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827672.759676:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827672.759686:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827672.759690:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827672.759692:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827672.759698:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4123 00000100:00080000:0.0:1787827672.759702:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827672.759706:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827672.759711:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827672.759712:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827672/real 1787827672] req@ffff9c1a788621c0 x1874668972828416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827672.759727:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972828800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827672.759731:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972828416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827672.759754:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827672.759758:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827672.759758:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827672.759762:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827672.759767:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827672.759774:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972828416:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827672.759775:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4123 00000100:00080000:0.0:1787827672.759782:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827672.759788:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827672.759793:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827672.759804:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972828928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827672.759807:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972828544:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827672.759823:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827672.759827:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827672.759829:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827672.759834:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4123 00000800:00000100:1.0:1787827672.759838:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827672.759840:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827672.759847:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827672.759853:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827672.759853:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827672/real 1787827672] req@ffff9c1a78860780 x1874668972828544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827672.759869:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972829056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827672.759870:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972828544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827672.759903:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827672.759904:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827672.759911:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827672.759912:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827672.759916:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827672.759921:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972828544:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827672.759925:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4123 00000100:00080000:0.0:1787827672.759932:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827672.759936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972828672:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827672.759939:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827672.759946:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827672.759960:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972829184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827672.759963:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827672.759976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827672/real 1787827672] req@ffff9c1a78861e00 x1874668972828672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827672.759986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972828672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827672.759994:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827672.759996:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827672.759999:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972828672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827672.760003:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972828800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827672.760014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827672.760021:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827672/real 1787827672] req@ffff9c1a78862580 x1874668972828800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827672.760028:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972828800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827672.760045:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827672.760047:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827672.760049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972828800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827672.760062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972828928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827672.760076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827672.760083:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827672/real 1787827672] req@ffff9c1a78861680 x1874668972828928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827672.760091:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972828928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827672.760096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827672.760098:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827672.760100:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972828928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827672.760103:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972829056:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827672.760113:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827672.760118:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827672/real 1787827672] req@ffff9c1a788630c0 x1874668972829056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827672.760126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972829056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827672.760131:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827672.760133:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827672.760135:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972829056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827672.760138:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972829184:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827672.760147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827672.760153:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827672/real 1787827672] req@ffff9c1a78863c00 x1874668972829184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827672.760160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972829184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827672.760165:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827672.760167:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827672.760168:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972829184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827673.271438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827673.271445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827673.527519:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827674.295419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827674.295419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827674.551522:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827675.319478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827675.319478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827675.575587:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827676.343498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827676.343506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827676.599485:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827677.367476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827677.367482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827677.623552:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827677.879456:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827677.879464:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827677.879468:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827677.879476:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4128 00000100:00080000:0.0:1787827677.879485:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827677.879490:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827677.879522:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972829312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827677.879545:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827677.879548:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827677.879549:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827677.879552:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4128 00000100:00080000:0.0:1787827677.879554:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827677.879558:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827677.879566:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827677.879576:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972829440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827677.879582:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827677.879585:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827677.879586:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827677.879588:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4128 00000100:00080000:0.0:1787827677.879589:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827677.879591:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827677.879594:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827677.879601:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972829568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827677.879609:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827677.879611:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827677.879612:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827677.879615:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4128 00000100:00080000:0.0:1787827677.879617:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827677.879619:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827677.879623:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827677.879628:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972829312:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827677.879633:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972829696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827677.879641:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827677.879644:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827677.879645:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827677.879648:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4128 00000100:00080000:0.0:1787827677.879650:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827677.879652:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827677.879655:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827677.879663:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972829824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827677.879669:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827677.879672:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827677.879672:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827677.879675:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4128 00000100:00080000:0.0:1787827677.879678:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827677.879681:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827677.879684:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827677.879690:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827677.879694:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972829952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827677.879701:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827677.879704:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827677.879706:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827677.879709:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4128 00000100:00080000:0.0:1787827677.879711:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827677.879713:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827677.879717:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827677.879726:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972830080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827677.879730:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827677/real 1787827677] req@ffff9c1a78860000 x1874668972829312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827677.879754:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972829312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827677.879762:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827677.879765:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827677.879768:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972829312:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827677.879774:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972829440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827677.879789:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827677.879796:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827677/real 1787827677] req@ffff9c1a78863c00 x1874668972829440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827677.879804:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972829440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827677.879809:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827677.879811:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827677.879813:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972829440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827677.879817:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972829568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827677.879827:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827677.879833:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827677/real 1787827677] req@ffff9c1a788630c0 x1874668972829568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827677.879841:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972829568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827677.879850:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827677.879852:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827677.879854:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972829568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827677.879872:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972829696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827677.879882:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827677.879888:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827677/real 1787827677] req@ffff9c1a78861680 x1874668972829696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827677.879895:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972829696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827677.879900:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827677.879902:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827677.879904:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972829696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827677.879907:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972829824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827677.879917:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827677.879922:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827677/real 1787827677] req@ffff9c1a78862580 x1874668972829824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827677.879930:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972829824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827677.879934:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827677.879936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827677.879939:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972829824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827677.879942:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972829952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827677.879953:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827677.879958:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827677/real 1787827677] req@ffff9c1a78861e00 x1874668972829952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827677.879965:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972829952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827677.879970:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827677.879972:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827677.879974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972829952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827677.879977:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972830080:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827677.879987:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827677.879993:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827677/real 1787827677] req@ffff9c1a78860780 x1874668972830080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827677.880018:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972830080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827677.880024:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827677.880026:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827677.880028:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972830080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827678.391453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827678.391453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827678.647520:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827679.415505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827679.415506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827679.671566:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827680.439466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827680.439474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827680.695534:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827681.463532:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827681.463544:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827681.719631:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827682.487398:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827682.487484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827682.743489:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827682.999455:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827682.999464:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827682.999468:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827682.999477:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4133 00000100:00080000:2.0:1787827682.999486:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827682.999492:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827682.999526:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972830208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827682.999551:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827682.999554:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827682.999561:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827682.999564:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4133 00000100:00080000:2.0:1787827682.999567:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827682.999570:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827682.999579:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827682.999591:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972830336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827682.999601:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827682.999604:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827682.999605:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827682.999611:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4133 00000100:00100000:1.0:1787827682.999613:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972830208:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827682.999614:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827682.999621:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827682.999640:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827682.999649:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972830464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827682.999658:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827682.999661:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827682.999662:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827682.999665:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4133 00000100:00080000:2.0:1787827682.999668:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827682.999670:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827682.999674:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827682.999699:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972830592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827682.999707:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827682.999721:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827682.999723:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827682.999725:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4133 00000100:00080000:2.0:1787827682.999729:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827682.999734:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787827682.999736:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787827682.999740:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827682.999753:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972830720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827682.999783:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827682.999785:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827682.999787:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827682.999794:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4133 00000100:00100000:1.0:1787827682.999795:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827682/real 1787827682] req@ffff9c1a49d8cb40 x1874668972830208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827682.999797:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827682.999805:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827682.999813:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827682.999823:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972830208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827682.999827:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972830848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827682.999876:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827682.999880:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827682.999882:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827682.999883:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827682.999892:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827682.999893:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4133 00000100:00080000:2.0:1787827682.999902:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827682.999905:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972830208:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827682.999911:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827682.999917:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827682.999927:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972830976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827682.999936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972830336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827682.999983:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827682.999997:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827682/real 1787827682] req@ffff9c1a49d8ed00 x1874668972830336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827683.000033:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972830336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827683.000039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827683.000042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827683.000044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972830336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827683.000049:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972830464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827683.000064:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827683.000070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827682/real 1787827683] req@ffff9c1a49d8c3c0 x1874668972830464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827683.000079:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972830464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827683.000084:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827683.000086:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827683.000088:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972830464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827683.000092:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972830592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827683.000102:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827683.000108:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827682/real 1787827683] req@ffff9c1a49d8e580 x1874668972830592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827683.000115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972830592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827683.000121:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827683.000122:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827683.000132:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972830592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827683.000139:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972830720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827683.000151:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827683.000158:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827682/real 1787827683] req@ffff9c1a49d8fc00 x1874668972830720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827683.000166:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972830720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827683.000176:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827683.000178:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827683.000181:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972830720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827683.000186:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972830848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827683.000242:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827683.000257:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827682/real 1787827683] req@ffff9c1a49d8e1c0 x1874668972830848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827683.000271:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972830848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827683.000277:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827683.000280:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827683.000282:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972830848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827683.000300:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972830976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827683.000317:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827683.000326:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827682/real 1787827683] req@ffff9c1a49d8d680 x1874668972830976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827683.000334:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972830976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827683.000341:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827683.000343:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827683.000375:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972830976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827683.511466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827683.511466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827683.767496:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827684.535449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827684.535449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827684.791497:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827685.559416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827685.559422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827685.815504:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827686.583447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827686.583452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827686.839503:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827687.607477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827687.607477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827687.863582:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827688.119469:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827688.119477:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827688.119481:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827688.119489:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4138 00000100:00080000:0.0:1787827688.119498:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827688.119504:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827688.119538:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972831104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827688.119561:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827688.119564:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827688.119566:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827688.119569:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4138 00000100:00080000:0.0:1787827688.119571:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827688.119573:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827688.119581:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827688.119590:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972831232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827688.119598:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827688.119601:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827688.119602:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827688.119606:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4138 00000100:00080000:0.0:1787827688.119609:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827688.119613:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827688.119616:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972831104:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787827688.119618:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827688.119629:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972831360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827688.119638:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827688.119641:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827688.119642:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827688.119645:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4138 00000100:00080000:0.0:1787827688.119648:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827688.119650:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827688.119653:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827688.119661:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972831488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827688.119667:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827688.119670:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827688.119671:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827688.119673:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4138 00000100:00080000:0.0:1787827688.119676:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827688.119680:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827688.119682:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827688.119687:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827688.119695:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972831616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827688.119702:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827688.119705:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827688.119706:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827688.119709:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4138 00000100:00080000:0.0:1787827688.119711:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827688.119715:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827688.119718:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827688.119721:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827688/real 1787827688] req@ffff9c1a788621c0 x1874668972831104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827688.119729:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972831744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827688.119739:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827688.119744:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787827688.119745:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972831104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827688.119746:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827688.119756:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4138 00000100:00080000:0.0:1787827688.119765:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827688.119765:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827688.119772:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827688.119775:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972831104:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827688.119783:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827688.119785:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972831232:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787827688.119793:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827688.119813:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827688.119814:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972831872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827688.119843:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827688/real 1787827688] req@ffff9c1a78862940 x1874668972831232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827688.119866:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972831232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827688.119873:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827688.119876:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827688.119879:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972831232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827688.119902:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972831360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827688.119916:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827688.119923:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827688/real 1787827688] req@ffff9c1a78860b40 x1874668972831360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827688.119948:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972831360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827688.119955:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827688.119957:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827688.119972:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972831360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827688.119976:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972831488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827688.119992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827688.120000:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827688/real 1787827688] req@ffff9c1a78861a40 x1874668972831488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827688.120008:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972831488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827688.120014:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827688.120016:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827688.120018:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972831488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827688.120022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972831616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827688.120033:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827688.120039:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827688/real 1787827688] req@ffff9c1a78860f00 x1874668972831616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827688.120048:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972831616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827688.120053:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827688.120055:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827688.120057:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972831616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827688.120061:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972831744:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827688.120072:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827688.120079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827688/real 1787827688] req@ffff9c1a78862d00 x1874668972831744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827688.120087:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972831744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827688.120092:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827688.120094:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827688.120096:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972831744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827688.120100:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972831872:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827688.120110:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827688.120116:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827688/real 1787827688] req@ffff9c1a788612c0 x1874668972831872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827688.120124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972831872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827688.120135:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827688.120137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827688.120139:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972831872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827688.631417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827688.631418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827688.887539:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827689.655449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827689.655450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827689.911522:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827690.679496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827690.679501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827690.935430:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827691.703434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827691.703436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827691.959515:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827692.727475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827692.727475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827692.983585:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827693.239482:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827693.239490:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827693.239494:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827693.239502:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4143 00000100:00080000:2.0:1787827693.239511:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827693.239517:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827693.239548:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972832000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827693.239570:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827693.239573:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827693.239574:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827693.239577:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4143 00000100:00080000:2.0:1787827693.239579:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827693.239581:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827693.239589:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827693.239597:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972832128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827693.239604:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827693.239607:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827693.239608:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827693.239610:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4143 00000100:00080000:2.0:1787827693.239612:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827693.239615:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827693.239620:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827693.239624:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972832000:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827693.239629:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972832256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827693.239638:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827693.239640:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827693.239641:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827693.239644:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4143 00000100:00080000:2.0:1787827693.239646:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827693.239648:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827693.239651:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827693.239660:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972832384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827693.239666:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827693.239668:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827693.239669:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827693.239672:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4143 00000100:00080000:2.0:1787827693.239675:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827693.239677:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827693.239680:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827693.239685:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827693.239689:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972832512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827693.239696:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827693.239698:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827693.239700:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827693.239704:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4143 00000100:00080000:2.0:1787827693.239707:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827693.239709:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827693.239713:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827693.239724:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827693/real 1787827693] req@ffff9c1a49d8f0c0 x1874668972832000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827693.239725:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972832640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827693.239754:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827693.239758:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827693.239759:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827693.239764:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4143 00000100:00100000:1.0:1787827693.239765:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972832000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827693.239767:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827693.239772:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827693.239780:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827693.239784:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827693.239790:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827693.239790:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972832768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827693.239794:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972832000:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827693.239808:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972832128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827693.239826:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827693.239834:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827693/real 1787827693] req@ffff9c1a49d8c780 x1874668972832128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827693.239842:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972832128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827693.239852:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827693.239854:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827693.239856:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972832128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827693.239874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972832256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827693.239885:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827693.239906:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827693/real 1787827693] req@ffff9c1a49d8de00 x1874668972832256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827693.239915:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972832256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827693.239921:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827693.239923:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827693.239925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972832256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827693.239930:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972832384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827693.239944:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827693.239951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827693/real 1787827693] req@ffff9c1a49d8c000 x1874668972832384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827693.239958:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972832384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827693.239963:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827693.239964:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827693.239967:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972832384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827693.239971:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972832512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827693.239980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827693.239986:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827693/real 1787827693] req@ffff9c1a49d8da40 x1874668972832512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827693.239993:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972832512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827693.239998:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827693.239999:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827693.240001:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972832512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827693.240005:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972832640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827693.240015:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827693.240020:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827693/real 1787827693] req@ffff9c1a49d8cf00 x1874668972832640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827693.240028:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972832640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827693.240032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827693.240034:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827693.240036:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972832640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827693.240039:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972832768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827693.240048:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827693.240053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827693/real 1787827693] req@ffff9c1a49d8d680 x1874668972832768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827693.240060:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972832768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827693.240066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827693.240067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827693.240069:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972832768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827693.751428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827693.751428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827694.007504:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827694.775489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827694.775489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827695.031575:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827695.799479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827695.799492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827696.056885:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827696.823462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827696.823462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827697.079516:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827697.847412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827697.847412:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827698.103438:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827698.359488:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827698.359509:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827698.359514:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827698.359525:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4148 00000100:00080000:0.0:1787827698.359536:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827698.359543:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827698.359583:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972832896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827698.359630:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827698.359634:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827698.359636:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827698.359641:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4148 00000100:00080000:0.0:1787827698.359645:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827698.359650:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827698.359661:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827698.359675:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972832896:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827698.359708:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972833024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827698.359721:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827698.359735:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827698.359737:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827698.359741:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4148 00000100:00080000:0.0:1787827698.359745:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827698.359765:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827698.359771:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827698.359784:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972833152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827698.359800:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827698.359805:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827698.359810:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827698.359812:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827698.359816:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4148 00000100:00080000:0.0:1787827698.359820:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827698.359824:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827698.359828:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827698.359840:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972833280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827698.359847:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827698.359851:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827698.359852:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827698.359873:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4148 00000100:00080000:0.0:1787827698.359894:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827698.359908:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827698.359908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827698/real 1787827698] req@ffff9c1a78863840 x1874668972832896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827698.359915:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827698.359929:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972833408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827698.359936:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827698.359940:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827698.359941:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972832896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827698.359942:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827698.359966:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4148 00000100:00080000:0.0:1787827698.359974:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827698.359981:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827698.359986:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827698.359986:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827698.359995:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827698.360001:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972833536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827698.360004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972832896:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827698.360019:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827698.360022:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827698.360024:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827698.360027:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4148 00000100:00080000:0.0:1787827698.360030:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827698.360033:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827698.360036:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827698.360037:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972833024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827698.360049:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972833664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827698.360070:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827698.360087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827698/real 1787827698] req@ffff9c1a788612c0 x1874668972833024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827698.360104:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972833024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827698.360126:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827698.360128:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827698.360131:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972833024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827698.360135:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972833152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827698.360148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827698.360154:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827698/real 1787827698] req@ffff9c1a78862d00 x1874668972833152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827698.360162:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972833152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827698.360167:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827698.360168:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827698.360170:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972833152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827698.360190:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972833280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827698.360204:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827698.360211:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827698/real 1787827698] req@ffff9c1a78860f00 x1874668972833280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827698.360220:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972833280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827698.360232:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827698.360234:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827698.360236:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972833280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827698.360240:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972833408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827698.360251:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827698.360258:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827698/real 1787827698] req@ffff9c1a78861a40 x1874668972833408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827698.360266:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972833408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827698.360272:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827698.360274:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827698.360277:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972833408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827698.360281:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972833536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827698.360305:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827698.360312:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827698/real 1787827698] req@ffff9c1a78860b40 x1874668972833536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827698.360321:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972833536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827698.360327:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827698.360345:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827698.360348:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972833536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827698.360395:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972833664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827698.360425:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827698.360435:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827698/real 1787827698] req@ffff9c1a78862940 x1874668972833664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827698.360445:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972833664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827698.360452:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827698.360455:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827698.360458:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972833664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827698.871439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827698.871439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827699.127559:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827699.895398:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827699.895414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827700.151487:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827700.919412:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827700.919416:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827701.175618:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827701.943416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827701.943431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827702.199490:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827702.967421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827702.967421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827703.223508:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827703.479487:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827703.479498:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827703.479503:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827703.479514:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4153 00000100:00080000:2.0:1787827703.479527:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827703.479548:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827703.479587:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972833792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827703.479621:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827703.479625:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827703.479627:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827703.479632:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4153 00000100:00080000:2.0:1787827703.479637:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827703.479641:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827703.479652:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827703.479672:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972833792:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827703.479672:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972833920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827703.479691:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827703.479695:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827703.479697:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827703.479702:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4153 00000100:00080000:2.0:1787827703.479707:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827703.479713:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827703.479720:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827703.479742:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972834048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827703.479752:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827703.479754:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827703.479756:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827703.479759:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4153 00000100:00080000:2.0:1787827703.479761:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827703.479764:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827703.479768:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827703.479777:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972834176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827703.479785:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827703.479788:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827703.479789:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827703.479791:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4153 00000100:00080000:2.0:1787827703.479794:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827703.479797:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827703.479801:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972833920:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787827703.479802:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827703.479817:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972834304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827703.479839:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827703.479842:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827703.479844:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827703.479847:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4153 00000100:00080000:2.0:1787827703.479850:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827703.479855:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972834048:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827703.479855:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827703.479864:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827703.479875:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972834432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827703.479883:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827703.479888:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972834176:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827703.479888:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827703.479892:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827703.479898:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4153 00000100:00080000:2.0:1787827703.479901:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827703.479904:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827703.479908:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827703.479918:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972834304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827703.479935:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972834432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827703.479939:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972834560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827703.479966:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972834560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827703.991422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827703.991424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827704.247606:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827705.015459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827705.015459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827705.271493:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827706.039435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827706.039435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827706.295566:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787827706.551602:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:3.0:1787827706.551680:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827706.551698:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827706.551733:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827706.551746:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827706.551757:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787827706.551769:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827703/real 1787827706] req@ffff9c1a49d8fc00 x1874668972833920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787827706.551769:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827706.551782:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787827706.551792:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787827706.551812:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972833920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827706.551827:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827706.551834:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827706.551839:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972833920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827706.551848:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827703/real 1787827706] req@ffff9c1a49d8e580 x1874668972834048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827706.551863:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972834048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827706.551871:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827706.551875:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827706.551879:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972834048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827706.551886:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827703/real 1787827706] req@ffff9c1a49d8c3c0 x1874668972834176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827706.551899:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972834176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827706.551909:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827706.551913:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827706.551917:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972834176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827706.551924:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827703/real 1787827706] req@ffff9c1a49d8e1c0 x1874668972833792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827706.551938:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972833792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827706.551947:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827706.551950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827706.551954:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972833792:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827706.552006:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827703/real 1787827706] req@ffff9c1a49d8d680 x1874668972834560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827706.552019:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972834560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827706.552026:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827706.552029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827706.552032:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972834560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827706.552038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827703/real 1787827706] req@ffff9c1a49d8ed00 x1874668972834304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827706.552048:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972834304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827706.552055:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827706.552058:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827706.552061:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972834304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827706.552067:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827703/real 1787827706] req@ffff9c1a49d8cb40 x1874668972834432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827706.552078:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972834432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827706.552085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827706.552088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827706.552092:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972834432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827707.063433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827707.063433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827707.319560:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827708.087456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827708.087456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827708.343557:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827708.599454:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827708.599463:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827708.599467:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827708.599495:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4158 00000100:00080000:2.0:1787827708.599507:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827708.599515:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827708.599558:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972834688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827708.599615:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827708.599619:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827708.599622:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827708.599628:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4158 00000100:00080000:2.0:1787827708.599632:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827708.599637:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827708.599648:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827708.599666:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972834816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827708.599681:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827708.599686:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827708.599688:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827708.599693:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4158 00000100:00080000:2.0:1787827708.599697:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827708.599702:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972834688:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827708.599707:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827708.599721:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827708.599735:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972834944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827708.599759:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827708.599764:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827708.599766:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827708.599771:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4158 00000100:00080000:2.0:1787827708.599806:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827708.599812:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787827708.599847:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787827708.599848:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827708.599868:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972835072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827708.599908:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827708.599913:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827708.599916:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827708.599926:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4158 00000100:00100000:0.0:1787827708.599932:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827708/real 1787827708] req@ffff9c1a49d8cf00 x1874668972834688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827708.599934:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827708.599947:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827708.599962:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827708.599999:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972834688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827708.600007:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972835200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827708.600058:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827708.600060:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827708.600073:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827708.600077:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827708.600079:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827708.600092:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4158 00000100:00100000:0.0:1787827708.600102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972834688:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827708.600104:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827708.600118:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827708.600128:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972834816:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787827708.600130:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827708.600158:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972835328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827708.600174:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827708.600179:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827708.600183:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827708.600192:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827708.600197:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4158 00000100:00080000:2.0:1787827708.600207:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827708.600215:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827708.600224:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827708.600233:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827708/real 1787827708] req@ffff9c1a49d8da40 x1874668972834816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827708.600252:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972835456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827708.600269:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972834816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827708.600290:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827708.600295:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827708.600303:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972834816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827708.600418:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972834944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827708.600458:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827708.600478:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827708/real 1787827708] req@ffff9c1a49d8c000 x1874668972834944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827708.600498:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972834944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827708.600511:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827708.600516:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827708.600521:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972834944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827708.600532:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972835072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827708.600565:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827708.600584:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827708/real 1787827708] req@ffff9c1a49d8de00 x1874668972835072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827708.600605:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972835072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827708.600633:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827708.600638:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827708.600644:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972835072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827708.600654:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972835200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827708.600689:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827708.600709:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827708/real 1787827708] req@ffff9c1a49d8c780 x1874668972835200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827708.600729:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972835200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827708.600743:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827708.600749:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827708.600754:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972835200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827708.600765:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972835328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827708.600826:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827708.600861:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827708/real 1787827708] req@ffff9c1a49d8f0c0 x1874668972835328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827708.600877:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972835328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827708.600887:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827708.600891:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827708.600896:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972835328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827708.600903:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972835456:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827708.600929:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827708.600944:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827708/real 1787827708] req@ffff9c1a49d8cb40 x1874668972835456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827708.600960:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972835456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827708.600969:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827708.600973:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827708.600978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972835456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827709.111447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827709.111561:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827709.367457:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827710.135483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827710.135490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827710.391515:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827711.159442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827711.159442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827711.415509:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827712.183461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827712.183461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827712.439563:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827713.207436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827713.207438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827713.463534:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827713.719466:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827713.719475:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827713.719479:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827713.719489:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4164 00000100:00080000:3.0:1787827713.719500:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827713.719508:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827713.719549:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972835584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827713.719573:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827713.719578:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827713.719579:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827713.719582:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4164 00000100:00080000:3.0:1787827713.719585:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827713.719588:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827713.719596:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827713.719607:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972835712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827713.719616:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827713.719618:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827713.719620:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827713.719623:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4164 00000100:00080000:3.0:1787827713.719625:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827713.719627:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827713.719630:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827713.719637:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972835840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827713.719648:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827713.719650:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972835584:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827713.719652:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827713.719655:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827713.719669:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4164 00000100:00080000:3.0:1787827713.719672:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827713.719674:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827713.719678:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827713.719686:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972835968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827713.719692:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827713.719695:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827713.719696:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827713.719699:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4164 00000100:00080000:3.0:1787827713.719702:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827713.719705:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827713.719708:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827713.719714:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972836096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827713.719720:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827713.719722:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827713.719723:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827713.719725:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4164 00000100:00080000:3.0:1787827713.719735:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827713.719739:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827713.719744:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827713.719762:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827713.719774:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972836224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827713.719783:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827713.719785:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827713.719787:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827713.719793:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4164 00000100:00080000:3.0:1787827713.719796:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827713.719800:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827713.719804:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827713.719813:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972836352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827713.719824:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827713/real 1787827713] req@ffff9c1a7895f840 x1874668972835584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827713.719845:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972835584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827713.719854:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827713.719858:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827713.719861:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972835584:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827713.719868:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972835712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827713.719892:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827713.719901:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827713/real 1787827713] req@ffff9c1a7895f0c0 x1874668972835712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827713.719910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972835712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827713.719915:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827713.719918:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827713.719920:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972835712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827713.719944:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972835840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827713.719956:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827713.719962:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827713/real 1787827713] req@ffff9c1a7895da40 x1874668972835840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827713.719970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972835840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827713.719975:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827713.719977:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827713.719979:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972835840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827713.719983:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972835968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827713.719994:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827713.720000:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827713/real 1787827713] req@ffff9c1a7895e1c0 x1874668972835968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827713.720012:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972835968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827713.720018:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827713.720020:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827713.720022:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972835968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827713.720025:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972836096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787827713.720036:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827713.720042:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827713/real 1787827713] req@ffff9c1a7895de00 x1874668972836096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827713.720050:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972836096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827713.720055:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827713.720057:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827713.720059:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972836096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827713.720063:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972836224:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827713.720074:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827713.720091:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827713/real 1787827713] req@ffff9c1a7895c780 x1874668972836224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827713.720102:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972836224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827713.720112:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827713.720114:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827713.720116:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972836224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827713.720121:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972836352:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787827713.720133:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827713.720139:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827713/real 1787827713] req@ffff9c1a7895cb40 x1874668972836352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827713.720146:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972836352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827713.720152:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827713.720154:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827713.720156:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972836352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827714.231492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827714.231496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827714.487504:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827715.255421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827715.255437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827715.511617:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827716.279563:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827716.279576:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827716.535520:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827717.303412:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827717.303413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827717.559495:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827718.327465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827718.327465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827718.583463:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827718.839512:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827718.839521:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827718.839525:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827718.839534:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4169 00000100:00080000:0.0:1787827718.839545:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827718.839552:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827718.839592:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972836480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827718.839618:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827718.839622:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827718.839624:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827718.839628:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4169 00000100:00080000:0.0:1787827718.839630:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827718.839633:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827718.839642:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827718.839653:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972836608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827718.839664:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827718.839667:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827718.839669:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827718.839673:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4169 00000100:00080000:0.0:1787827718.839678:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827718.839680:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972836480:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827718.839684:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827718.839692:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827718.839701:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972836736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827718.839709:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827718.839712:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827718.839729:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827718.839732:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4169 00000100:00080000:0.0:1787827718.839735:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827718.839737:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827718.839742:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827718.839755:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972836864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827718.839763:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827718.839766:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827718.839768:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827718.839771:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4169 00000100:00080000:0.0:1787827718.839774:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827718.839777:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827718.839780:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827718.839788:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827718.839793:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972836992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827718.839802:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827718.839805:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827718.839806:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827718.839810:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4169 00000100:00080000:0.0:1787827718.839814:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827718.839817:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827718.839821:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827718.839831:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972837120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827718.839838:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827718.839841:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827718.839844:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827718.839848:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4169 00000100:00080000:0.0:1787827718.839852:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827718.839857:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827718/real 1787827718] req@ffff9c1a788621c0 x1874668972836480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827718.839858:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827718.839865:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827718.839877:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972837248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827718.839899:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972836480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827718.839917:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827718.839924:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827718.839929:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972836480:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827718.839958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972836608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827718.839989:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827718.840005:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827718/real 1787827718] req@ffff9c1a78860780 x1874668972836608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827718.840023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972836608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827718.840035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827718.840039:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827718.840044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972836608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827718.840052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972836736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827718.840076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827718.840090:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827718/real 1787827718] req@ffff9c1a78861e00 x1874668972836736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827718.840107:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972836736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827718.840127:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827718.840132:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827718.840137:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972836736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827718.840145:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972836864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827718.840168:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827718.840181:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827718/real 1787827718] req@ffff9c1a78862580 x1874668972836864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827718.840198:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972836864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827718.840210:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827718.840214:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827718.840219:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972836864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827718.840227:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972836992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827718.840250:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827718.840263:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827718/real 1787827718] req@ffff9c1a78861680 x1874668972836992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827718.840280:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972836992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827718.840291:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827718.840296:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827718.840301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972836992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827718.840309:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972837120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827718.840331:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827718.840381:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827718/real 1787827718] req@ffff9c1a788630c0 x1874668972837120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827718.840411:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972837120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827718.840423:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827718.840427:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827718.840432:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972837120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827718.840440:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972837248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827718.840466:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827718.840481:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827718/real 1787827718] req@ffff9c1a78863c00 x1874668972837248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827718.840497:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972837248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827718.840508:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827718.840513:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827718.840517:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972837248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827719.351408:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827719.351412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827719.607415:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827720.375464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827720.375476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827720.631450:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827721.399389:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827721.399404:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827721.655504:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827722.423490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827722.423490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827722.679527:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827723.447440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827723.447470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827723.703482:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827723.959464:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827723.959475:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827723.959480:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827723.959490:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4174 00000100:00080000:2.0:1787827723.959502:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827723.959510:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827723.959548:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972837376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827723.959576:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827723.959580:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827723.959582:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827723.959587:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4174 00000100:00080000:2.0:1787827723.959591:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827723.959596:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827723.959608:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972837376:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787827723.959608:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827723.959626:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972837504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827723.959639:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827723.959643:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827723.959646:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827723.959650:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4174 00000100:00080000:2.0:1787827723.959656:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827723.959660:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827723.959666:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827723.959670:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827723.959697:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972837632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827723.959711:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827723.959715:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827723.959717:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827723.959721:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4174 00000100:00100000:1.0:1787827723.959726:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827723/real 1787827723] req@ffff9c1a49d8ed00 x1874668972837376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827723.959728:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827723.959737:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827723.959745:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827723.959748:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972837376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827723.959759:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827723.959762:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972837760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827723.959764:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827723.959771:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972837376:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827723.959785:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827723.959789:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827723.959791:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827723.959796:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4174 00000100:00100000:1.0:1787827723.959799:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972837504:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827723.959802:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827723.959809:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827723.959815:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827723.959820:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827723.959832:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827723/real 1787827723] req@ffff9c1a49d8d680 x1874668972837504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827723.959838:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972837888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827723.959845:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972837504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827723.959871:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827723.959877:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827723.959880:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827723.959884:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972837504:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827723.959886:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827723.959891:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827723.959897:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972837632:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827723.959900:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4174 00000100:00080000:2.0:1787827723.959907:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827723.959913:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827723.959919:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827723.959923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827723.959939:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827723/real 1787827723] req@ffff9c1a49d8e1c0 x1874668972837632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827723.959939:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972838016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827723.959976:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827723.959977:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972837632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827723.959985:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827723.959990:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827723.959991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827723.959998:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827723.960001:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4174 00000100:00100000:1.0:1787827723.960004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972837632:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827723.960009:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827723.960011:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972837760:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827723.960017:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827723.960023:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827723.960029:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827723.960044:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827723/real 1787827723] req@ffff9c1a49d8c3c0 x1874668972837760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827723.960044:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972838144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827723.960073:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972837760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827723.960081:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827723.960084:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827723.960086:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972837760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827723.960099:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972837888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827723.960111:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827723.960117:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827723/real 1787827723] req@ffff9c1a49d8e580 x1874668972837888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827723.960124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972837888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827723.960130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827723.960132:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827723.960134:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972837888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827723.960137:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972838016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827723.960147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827723.960153:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827723/real 1787827723] req@ffff9c1a49d8fc00 x1874668972838016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827723.960160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972838016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827723.960165:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827723.960167:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827723.960169:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972838016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827723.960172:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972838144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827723.960181:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827723.960186:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827723/real 1787827723] req@ffff9c1a49d8ed00 x1874668972838144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827723.960193:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972838144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827723.960198:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827723.960200:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827723.960202:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972838144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827724.471487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827724.471505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827724.727627:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827725.495474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827725.495474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827725.751485:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827726.519462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827726.519462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827726.775485:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827727.543472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827727.543483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827727.799532:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827728.567430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827728.567430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827728.823538:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827729.079529:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827729.079536:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827729.079540:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827729.079549:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4179 00000100:00080000:3.0:1787827729.079559:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827729.079565:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827729.079600:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972838272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827729.079624:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827729.079627:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827729.079629:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827729.079634:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4179 00000100:00080000:3.0:1787827729.079637:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827729.079655:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827729.079667:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827729.079681:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972838400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827729.079683:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972838272:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827729.079692:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827729.079694:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827729.079696:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827729.079709:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4179 00000100:00080000:3.0:1787827729.079712:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827729.079715:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827729.079720:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827729.079729:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972838528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827729.079738:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827729.079740:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827729.079742:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827729.079745:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4179 00000100:00080000:3.0:1787827729.079748:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827729.079752:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827729.079752:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827729.079756:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827729.079764:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972838656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827729.079772:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827729.079774:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827729.079777:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827729.079780:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4179 00000100:00080000:3.0:1787827729.079783:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827729.079786:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827729.079790:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827729.079792:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827729/real 1787827729] req@ffff9c1a7895d2c0 x1874668972838272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827729.079799:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972838784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827729.079807:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827729.079810:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827729.079811:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972838272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827729.079812:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827729.079816:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4179 00000100:00080000:3.0:1787827729.079819:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827729.079822:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827729.079822:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827729.079826:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827729.079835:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827729.079836:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972838912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827729.079840:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972838272:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827729.079845:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827729.079848:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827729.079850:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827729.079853:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4179 00000100:00080000:3.0:1787827729.079856:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827729.079859:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827729.079863:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972838400:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787827729.079864:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827729.079873:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972839040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827729.079884:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827729.079897:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827729/real 1787827729] req@ffff9c1a7895cb40 x1874668972838400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827729.079918:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972838400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827729.079927:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827729.079931:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827729.079934:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972838400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827729.079941:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972838528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827729.079960:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827729.079979:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827729/real 1787827729] req@ffff9c1a7895c780 x1874668972838528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827729.079987:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972838528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827729.079992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827729.079994:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827729.079996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972838528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827729.080000:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972838656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827729.080010:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827729.080016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827729/real 1787827729] req@ffff9c1a7895de00 x1874668972838656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827729.080023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972838656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827729.080029:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827729.080031:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827729.080033:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972838656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827729.080036:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972838784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827729.080046:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827729.080051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827729/real 1787827729] req@ffff9c1a7895e1c0 x1874668972838784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827729.080058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972838784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827729.080064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827729.080065:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827729.080067:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972838784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827729.080070:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972838912:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827729.080081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827729.080086:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827729/real 1787827729] req@ffff9c1a7895da40 x1874668972838912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827729.080101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972838912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827729.080106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827729.080109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827729.080111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972838912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827729.080133:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972839040:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827729.080151:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827729.080157:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827729/real 1787827729] req@ffff9c1a7895f0c0 x1874668972839040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827729.080170:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972839040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827729.080175:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827729.080177:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827729.080179:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972839040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827729.591446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827729.591447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827729.847491:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827730.615409:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827730.615414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827730.871557:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827731.639472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827731.639472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827731.895506:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827732.663487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827732.663611:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827732.919641:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827733.687460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827733.687473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827733.943526:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827734.199434:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827734.199442:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827734.199446:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827734.199454:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4184 00000100:00080000:3.0:1787827734.199464:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827734.199470:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827734.199504:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972839168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827734.199531:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827734.199534:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827734.199535:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827734.199538:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4184 00000100:00080000:3.0:1787827734.199540:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827734.199542:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827734.199550:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827734.199557:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972839296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827734.199564:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827734.199566:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827734.199567:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827734.199569:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4184 00000100:00080000:3.0:1787827734.199571:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827734.199572:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827734.199575:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827734.199580:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972839424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827734.199587:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827734.199590:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827734.199591:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827734.199593:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4184 00000100:00080000:3.0:1787827734.199596:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827734.199599:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827734.199602:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827734.199622:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972839168:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827734.199627:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972839552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827734.199636:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827734.199638:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827734.199640:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827734.199642:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4184 00000100:00080000:3.0:1787827734.199645:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827734.199648:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827734.199651:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827734.199661:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972839680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827734.199667:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827734.199669:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827734.199670:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827734.199673:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4184 00000100:00080000:3.0:1787827734.199677:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827734.199679:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827734.199682:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827734.199687:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827734.199692:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972839808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827734.199703:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827734.199706:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827734.199707:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827734.199709:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4184 00000100:00080000:3.0:1787827734.199712:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827734.199714:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827734.199718:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827734.199726:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827734/real 1787827734] req@ffff9c1a7895f840 x1874668972839168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827734.199728:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972839936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827734.199767:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972839168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827734.199776:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827734.199780:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827734.199784:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972839168:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827734.199788:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972839296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827734.199803:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827734.199811:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827734/real 1787827734] req@ffff9c1a7895c000 x1874668972839296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827734.199819:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972839296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827734.199825:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827734.199827:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827734.199830:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972839296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827734.199833:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972839424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827734.199843:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827734.199849:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827734/real 1787827734] req@ffff9c1a7895f480 x1874668972839424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827734.199855:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972839424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827734.199861:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827734.199863:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827734.199864:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972839424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827734.199885:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972839552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827734.199897:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827734.199903:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827734/real 1787827734] req@ffff9c1a7895cf00 x1874668972839552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827734.199910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972839552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827734.199915:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827734.199917:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827734.199919:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972839552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827734.199923:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972839680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827734.199933:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827734.199939:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827734/real 1787827734] req@ffff9c1a7895c3c0 x1874668972839680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827734.199947:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972839680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827734.199953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827734.199955:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827734.199958:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972839680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827734.199964:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972839808:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827734.199982:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827734.199993:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827734/real 1787827734] req@ffff9c1a7895fc00 x1874668972839808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827734.200007:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972839808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827734.200016:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827734.200019:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827734.200023:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972839808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827734.200028:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972839936:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827734.200044:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827734.200052:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827734/real 1787827734] req@ffff9c1a7895e580 x1874668972839936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827734.200060:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972839936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827734.200066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827734.200070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827734.200073:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972839936:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827734.711406:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827734.711413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827734.967593:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827735.735447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827735.735447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827735.991527:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827736.759417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827736.759417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827737.015482:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827737.783507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827737.783508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827738.039570:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827738.807454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827738.807482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827739.063662:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827739.319418:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827739.319426:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827739.319430:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827739.319439:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4189 00000100:00080000:3.0:1787827739.319448:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827739.319455:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827739.319488:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972840064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827739.319512:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827739.319515:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827739.319517:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827739.319520:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4189 00000100:00080000:3.0:1787827739.319522:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827739.319525:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827739.319532:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827739.319542:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972840192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827739.319549:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827739.319551:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827739.319552:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827739.319554:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4189 00000100:00080000:3.0:1787827739.319556:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827739.319558:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827739.319563:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827739.319583:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972840320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827739.319592:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827739.319594:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827739.319596:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827739.319600:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4189 00000100:00080000:3.0:1787827739.319604:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827739.319608:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827739.319632:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972840064:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787827739.319636:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827739.319663:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972840448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827739.319682:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827739.319687:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827739.319689:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827739.319697:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4189 00000100:00080000:3.0:1787827739.319701:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827739.319706:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827739.319714:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827739.319727:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972840576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827739.319738:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827739.319747:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827739.319747:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827739.319756:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827739.319766:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4189 00000100:00080000:3.0:1787827739.319771:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827739.319778:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827739.319785:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827739.319804:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972840704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827739.319816:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827739/real 1787827739] req@ffff9c1a7895e940 x1874668972840064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827739.319820:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827739.319840:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827739.319851:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827739.319861:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4189 00000100:00100000:1.0:1787827739.319879:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972840064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827739.319885:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827739.319897:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827739.319906:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827739.319907:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827739.319922:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827739.319933:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972840064:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827739.319934:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972840832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827739.319953:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972840192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827739.319983:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827739.319999:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827739/real 1787827739] req@ffff9c1a7895e580 x1874668972840192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827739.320009:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972840192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827739.320017:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827739.320020:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827739.320023:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972840192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827739.320031:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972840320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827739.320051:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827739.320059:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827739/real 1787827739] req@ffff9c1a7895fc00 x1874668972840320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827739.320069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972840320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827739.320075:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827739.320078:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827739.320080:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972840320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827739.320117:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972840448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827739.320131:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827739.320154:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827739/real 1787827739] req@ffff9c1a7895c3c0 x1874668972840448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827739.320169:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972840448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827739.320182:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827739.320186:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827739.320191:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972840448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827739.320198:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972840576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827739.320213:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827739.320233:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827739/real 1787827739] req@ffff9c1a7895cf00 x1874668972840576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827739.320243:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972840576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827739.320257:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827739.320263:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827739.320268:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972840576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827739.320276:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972840704:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827739.320298:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827739.320308:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827739/real 1787827739] req@ffff9c1a7895f480 x1874668972840704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827739.320321:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972840704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827739.320328:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827739.320333:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827739.320337:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972840704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827739.320342:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972840832:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827739.320400:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827739.320409:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827739/real 1787827739] req@ffff9c1a7895c000 x1874668972840832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827739.320421:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972840832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827739.320428:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827739.320431:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827739.320435:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972840832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827739.831451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827739.831451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827740.087540:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827740.855414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827740.855418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827741.111447:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827741.879435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827741.879435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827742.135641:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827742.903444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827742.903446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827743.159455:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827743.927445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827743.927446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827744.183491:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827744.439503:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827744.439513:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827744.439518:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827744.439528:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4194 00000100:00080000:0.0:1787827744.439569:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827744.439576:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827744.439617:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972840960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827744.439644:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827744.439647:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827744.439649:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827744.439653:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4194 00000100:00080000:0.0:1787827744.439656:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827744.439658:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827744.439668:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827744.439682:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972841088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827744.439709:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827744.439712:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972840960:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827744.439713:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827744.439717:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827744.439725:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4194 00000100:00080000:0.0:1787827744.439728:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827744.439731:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827744.439735:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827744.439747:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972841216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827744.439755:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827744.439758:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827744.439760:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827744.439763:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4194 00000100:00080000:0.0:1787827744.439768:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827744.439772:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827744.439775:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827744.439782:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827744.439787:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972841344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827744.439796:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827744.439799:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827744.439800:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827744.439805:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4194 00000100:00080000:0.0:1787827744.439808:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827744.439811:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827744.439815:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827744.439826:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827744/real 1787827744] req@ffff9c1a78860000 x1874668972840960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827744.439827:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972841472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827744.439848:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827744.439850:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827744.439851:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827744.439856:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4194 00000100:00080000:0.0:1787827744.439862:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827744.439868:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827744.439872:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827744.439877:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972840960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827744.439882:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972841600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827744.439908:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827744.439908:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827744.439913:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827744.439914:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827744.439931:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4194 00000100:00080000:0.0:1787827744.439933:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827744.439936:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827744.439936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787827744.439943:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827744.439950:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972840960:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827744.439955:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972841728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827744.440008:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972841088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827744.440040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827744.440054:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827744/real 1787827744] req@ffff9c1a78863c00 x1874668972841088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827744.440065:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972841088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827744.440073:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827744.440076:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827744.440079:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972841088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827744.440085:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972841216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827744.440100:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827744.440108:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827744/real 1787827744] req@ffff9c1a788630c0 x1874668972841216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827744.440117:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972841216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827744.440124:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827744.440127:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827744.440129:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972841216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827744.440134:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972841344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827744.440147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827744.440155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827744/real 1787827744] req@ffff9c1a78861680 x1874668972841344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827744.440171:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972841344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827744.440178:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827744.440180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827744.440183:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972841344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827744.440187:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972841472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827744.440212:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827744.440225:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827744/real 1787827744] req@ffff9c1a78862580 x1874668972841472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827744.440241:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972841472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827744.440252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827744.440257:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827744.440261:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972841472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827744.440270:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972841600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827744.440305:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827744.440315:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827744/real 1787827744] req@ffff9c1a78861e00 x1874668972841600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827744.440327:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972841600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827744.440357:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827744.440399:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827744.440414:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972841600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827744.440421:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972841728:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827744.440443:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827744.440454:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827744/real 1787827744] req@ffff9c1a78860780 x1874668972841728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827744.440464:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972841728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827744.440472:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827744.440475:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827744.440477:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972841728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827744.951437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827744.951475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827745.207510:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827745.975415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827745.975420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827746.231571:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827746.999461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827746.999461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827747.255534:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827748.023461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827748.023461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827748.279499:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827749.047492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827749.047492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827749.303517:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827749.559413:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827749.559421:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827749.559426:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827749.559435:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4199 00000100:00080000:0.0:1787827749.559446:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827749.559452:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827749.559486:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972841856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827749.559523:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827749.559527:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827749.559529:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827749.559532:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4199 00000100:00080000:0.0:1787827749.559535:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827749.559555:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827749.559563:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827749.559575:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972841984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827749.559580:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972841856:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827749.559589:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827749.559593:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827749.559594:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827749.559597:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4199 00000100:00080000:0.0:1787827749.559601:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827749.559604:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827749.559607:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827749.559617:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972842112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827749.559624:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827749.559627:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827749.559629:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827749.559632:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4199 00000100:00080000:0.0:1787827749.559634:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827749.559638:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787827749.559642:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787827749.559644:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827749.559656:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972842240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827749.559662:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827749.559664:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827749.559665:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827749.559668:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4199 00000100:00080000:0.0:1787827749.559671:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827749.559673:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827749.559678:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827749.559678:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827749/real 1787827749] req@ffff9c1a788621c0 x1874668972841856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827749.559690:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972842368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827749.559697:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827749.559698:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972841856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827749.559702:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827749.559705:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827749.559714:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4199 00000100:00080000:0.0:1787827749.559719:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827749.559725:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827749.559732:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827749.559733:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827749.559739:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827749.559743:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972842496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827749.559745:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972841856:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827749.559759:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827749.559761:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827749.559763:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827749.559765:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4199 00000100:00080000:0.0:1787827749.559767:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827749.559770:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827749.559773:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827749.559780:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972842624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827749.559783:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972841984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827749.559813:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827749.559825:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827749/real 1787827749] req@ffff9c1a78862940 x1874668972841984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827749.559833:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972841984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827749.559839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827749.559842:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827749.559844:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972841984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827749.559848:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972842112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827749.559860:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827749.559865:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827749/real 1787827749] req@ffff9c1a78860b40 x1874668972842112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827749.559873:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972842112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827749.559878:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827749.559881:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827749.559883:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972842112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827749.559887:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972842240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827749.559896:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827749.559902:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827749/real 1787827749] req@ffff9c1a78861a40 x1874668972842240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827749.559910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972842240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827749.559915:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827749.559917:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827749.559918:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972842240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827749.559922:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972842368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827749.559931:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827749.559936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827749/real 1787827749] req@ffff9c1a78860f00 x1874668972842368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827749.559944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972842368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827749.559949:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827749.559950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827749.559952:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972842368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827749.559955:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972842496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827749.559966:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827749.559971:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827749/real 1787827749] req@ffff9c1a78862d00 x1874668972842496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827749.559979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972842496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827749.559984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827749.559986:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827749.559987:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972842496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827749.559999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972842624:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827749.560009:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827749.560014:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827749/real 1787827749] req@ffff9c1a788612c0 x1874668972842624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827749.560021:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972842624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827749.560027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827749.560028:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827749.560030:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972842624:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827750.071474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827750.071513:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827750.327528:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827751.095550:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827751.095550:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827751.351482:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827752.119407:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827752.119414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827752.375529:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827753.143455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827753.143455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827753.399527:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827754.167416:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827754.167416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827754.423525:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827754.679458:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827754.679482:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827754.679485:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827754.679493:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4205 00000100:00080000:2.0:1787827754.679502:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827754.679508:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827754.679539:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972842752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827754.679560:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827754.679563:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827754.679564:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827754.679567:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4205 00000100:00080000:2.0:1787827754.679569:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827754.679572:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827754.679580:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827754.679589:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972842880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827754.679597:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827754.679599:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972842752:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827754.679601:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827754.679604:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827754.679610:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4205 00000100:00080000:2.0:1787827754.679612:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827754.679614:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827754.679617:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827754.679626:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972843008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827754.679633:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827754.679636:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827754.679638:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827754.679641:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4205 00000100:00080000:2.0:1787827754.679643:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827754.679645:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827754.679648:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827754.679656:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972843136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827754.679661:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827754.679669:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827754.679672:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827754.679673:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827754.679677:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4205 00000100:00080000:2.0:1787827754.679682:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827754.679684:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827754.679687:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827754.679697:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972843264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827754.679702:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827754/real 1787827754] req@ffff9c1a49d8cb40 x1874668972842752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827754.679707:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827754.679714:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827754.679717:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827754.679724:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4205 00000100:00080000:2.0:1787827754.679730:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827754.679735:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827754.679739:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827754.679743:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972842752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827754.679749:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972843392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827754.679756:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827754.679758:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827754.679764:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827754.679766:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827754.679767:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827754.679771:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4205 00000100:00100000:1.0:1787827754.679778:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972842752:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827754.679782:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827754.679787:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827754.679790:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827754.679798:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972843520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827754.679809:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972842880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827754.679833:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827754.679844:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827754/real 1787827754] req@ffff9c1a49d8f0c0 x1874668972842880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827754.679853:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972842880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827754.679860:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827754.679864:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827754.679867:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972842880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827754.679874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972843008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827754.679894:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827754.679906:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827754/real 1787827754] req@ffff9c1a49d8c780 x1874668972843008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827754.679917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972843008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827754.679927:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827754.679929:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827754.679932:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972843008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827754.679936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972843136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827754.679951:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827754.679958:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827754/real 1787827754] req@ffff9c1a49d8de00 x1874668972843136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827754.679966:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972843136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827754.679972:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827754.679974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827754.679976:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972843136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827754.679979:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972843264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827754.679989:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827754.679994:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827754/real 1787827754] req@ffff9c1a49d8c000 x1874668972843264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827754.680001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972843264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827754.680006:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827754.680008:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827754.680011:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972843264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827754.680014:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972843392:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827754.680024:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827754.680030:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827754/real 1787827754] req@ffff9c1a49d8da40 x1874668972843392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827754.680036:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972843392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827754.680042:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827754.680043:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827754.680045:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972843392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827754.680060:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972843520:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827754.680070:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827754.680076:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827754/real 1787827754] req@ffff9c1a49d8cf00 x1874668972843520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827754.680083:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972843520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827754.680089:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827754.680091:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827754.680093:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972843520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827755.191457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827755.191457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827755.447493:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827756.215410:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827756.215410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827756.471474:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827757.239471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827757.239476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827757.495554:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827758.263423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827758.263429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827758.519510:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827759.287478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827759.287498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827759.543614:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827759.799500:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827759.799510:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827759.799515:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827759.799525:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4210 00000100:00080000:2.0:1787827759.799537:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827759.799544:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827759.799586:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972843648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827759.799634:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827759.799638:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827759.799640:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827759.799644:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4210 00000100:00080000:2.0:1787827759.799648:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827759.799652:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827759.799661:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827759.799675:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972843776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827759.799684:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972843648:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827759.799692:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827759.799697:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827759.799699:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827759.799705:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4210 00000100:00080000:2.0:1787827759.799709:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827759.799714:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827759.799719:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827759.799733:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972843904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827759.799744:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827759.799746:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827759.799751:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827759.799753:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827759.799759:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4210 00000100:00080000:2.0:1787827759.799765:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827759.799769:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827759.799775:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827759.799782:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827759/real 1787827759] req@ffff9c1a49d8cf00 x1874668972843648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827759.799791:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972844032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827759.799800:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972843648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827759.799809:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827759.799813:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827759.799814:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827759.799820:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827759.799820:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827759.799826:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972843648:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827759.799830:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4210 00000100:00080000:2.0:1787827759.799837:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827759.799841:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827759.799847:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827759.799849:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972843776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827759.799866:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972844160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827759.799873:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827759.799886:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827759.799887:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827759/real 1787827759] req@ffff9c1a49d8da40 x1874668972843776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827759.799895:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827759.799908:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827759.799918:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4210 00000100:00100000:1.0:1787827759.799920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972843776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827759.799924:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827759.799931:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827759.799935:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827759.799936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827759.799943:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972843776:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787827759.799946:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827759.799949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972843904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827759.799965:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972844288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827759.799969:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827759.799979:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827759/real 1787827759] req@ffff9c1a49d8c000 x1874668972843904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827759.800007:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827759.800008:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972843904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827759.800017:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827759.800021:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827759.800023:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827759.800027:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827759.800042:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4210 00000100:00100000:1.0:1787827759.800044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972843904:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827759.800050:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827759.800055:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972844032:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827759.800057:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827759.800067:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827759.800083:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972844416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827759.800089:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827759.800109:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827759/real 1787827759] req@ffff9c1a49d8de00 x1874668972844032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827759.800124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972844032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827759.800134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827759.800138:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827759.800142:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972844032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827759.800160:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972844160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827759.800176:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827759.800184:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827759/real 1787827759] req@ffff9c1a49d8c780 x1874668972844160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827759.800192:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972844160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827759.800202:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827759.800204:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827759.800207:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972844160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827759.800210:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972844288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827759.800221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827759.800227:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827759/real 1787827759] req@ffff9c1a49d8f0c0 x1874668972844288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827759.800234:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972844288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827759.800239:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827759.800240:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827759.800242:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972844288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827759.800246:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972844416:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827759.800255:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827759.800261:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827759/real 1787827759] req@ffff9c1a49d8cb40 x1874668972844416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827759.800268:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972844416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827759.800273:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827759.800274:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827759.800277:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972844416:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827760.311444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827760.311514:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827760.567486:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827761.335473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827761.335473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827761.591570:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827762.359447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827762.359447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827762.615480:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827763.383429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827763.383433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827763.639533:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827764.407501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827764.407509:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827764.663528:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827764.919507:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827764.919516:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827764.919520:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827764.919543:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4215 00000100:00080000:2.0:1787827764.919552:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827764.919558:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827764.919593:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972844544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827764.919616:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827764.919619:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827764.919621:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827764.919623:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4215 00000100:00080000:2.0:1787827764.919626:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827764.919628:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827764.919636:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827764.919644:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972844672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827764.919652:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827764.919654:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827764.919656:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827764.919658:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4215 00000100:00080000:2.0:1787827764.919661:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827764.919663:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827764.919666:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827764.919673:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972844544:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827764.919674:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972844800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827764.919687:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827764.919689:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827764.919691:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827764.919693:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4215 00000100:00080000:2.0:1787827764.919695:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827764.919697:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827764.919701:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827764.919711:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972844928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827764.919717:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827764.919720:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827764.919721:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827764.919724:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4215 00000100:00080000:2.0:1787827764.919726:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827764.919728:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827764.919731:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827764.919735:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827764.919739:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972845056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827764.919746:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827764.919748:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827764.919750:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827764.919753:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4215 00000100:00080000:2.0:1787827764.919755:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827764.919758:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827764.919761:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827764.919780:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972845184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827764.919786:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827764.919787:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827764/real 1787827764] req@ffff9c1a49d8ed00 x1874668972844544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827764.919801:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827764.919805:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827764.919812:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972844544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827764.919812:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4215 00000100:00080000:2.0:1787827764.919820:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827764.919827:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827764.919828:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827764.919844:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827764.919848:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972844544:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787827764.919849:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827764.919858:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972844672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827764.919870:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972845312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827764.919887:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827764.919905:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827764/real 1787827764] req@ffff9c1a49d8fc00 x1874668972844672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827764.919929:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972844672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827764.919937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827764.919940:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827764.919942:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972844672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827764.919965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972844800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827764.919981:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827764.919989:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827764/real 1787827764] req@ffff9c1a49d8e580 x1874668972844800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827764.919998:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972844800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827764.920003:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827764.920015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827764.920017:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972844800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827764.920021:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972844928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827764.920031:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827764.920037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827764/real 1787827764] req@ffff9c1a49d8c3c0 x1874668972844928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827764.920045:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972844928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827764.920050:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827764.920052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827764.920054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972844928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827764.920058:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972845056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827764.920067:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827764.920073:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827764/real 1787827764] req@ffff9c1a49d8e1c0 x1874668972845056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827764.920080:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972845056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827764.920086:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827764.920088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827764.920090:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972845056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827764.920093:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972845184:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827764.920103:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827764.920108:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827764/real 1787827764] req@ffff9c1a49d8d680 x1874668972845184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827764.920115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972845184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827764.920120:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827764.920121:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827764.920123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972845184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827764.920127:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972845312:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827764.920136:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827764.920141:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827764/real 1787827764] req@ffff9c1a49d8cb40 x1874668972845312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827764.920166:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972845312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827764.920178:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827764.920181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827764.920184:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972845312:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827765.431494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827765.431499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827765.687531:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827766.455508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827766.455509:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827766.711525:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827767.479489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827767.479489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827767.735573:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827768.503462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827768.503462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827768.759542:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827769.527444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827769.527444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827769.783491:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827770.039515:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827770.039523:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827770.039527:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827770.039535:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4220 00000100:00080000:0.0:1787827770.039544:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827770.039550:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827770.039585:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972845440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827770.039608:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827770.039610:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827770.039612:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827770.039615:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4220 00000100:00080000:0.0:1787827770.039617:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827770.039620:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827770.039627:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827770.039637:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972845568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827770.039645:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827770.039647:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827770.039649:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827770.039652:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4220 00000100:00080000:0.0:1787827770.039655:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827770.039657:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827770.039666:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827770.039669:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972845440:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827770.039676:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972845696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827770.039685:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827770.039687:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827770.039689:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827770.039691:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4220 00000100:00080000:0.0:1787827770.039694:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827770.039696:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827770.039699:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827770.039708:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972845824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827770.039714:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827770.039716:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827770.039718:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827770.039720:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4220 00000100:00080000:0.0:1787827770.039723:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827770.039725:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827770.039728:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827770.039735:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972845952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827770.039742:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827770.039745:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827770.039746:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827770.039748:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4220 00000100:00080000:0.0:1787827770.039751:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827770.039755:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827770.039760:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827770.039766:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972845568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827770.039776:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972846080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827770.039797:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827770.039801:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972845696:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827770.039803:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827770.039809:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827770.039814:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4220 00000100:00080000:0.0:1787827770.039819:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827770.039822:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972845824:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827770.039827:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827770.039833:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827770.039845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972845952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827770.039846:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972846208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827770.039869:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972846080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827770.039890:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972846208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827770.551469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827770.551470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827770.807531:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827771.575421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827771.575425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827771.831538:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827772.599475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827772.599475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827772.855571:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827773.111648:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787827773.111746:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827773.111761:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827773.111825:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827773.111834:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827773.111842:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827773.111849:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827773.111874:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827773.111885:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787827773.111921:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827770/real 1787827773] req@ffff9c1a78862940 x1874668972846208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827773.111954:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972846208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827773.111968:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827773.111974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827773.111981:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972846208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827773.111990:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827770/real 1787827773] req@ffff9c1a78862d00 x1874668972845696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827773.112005:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972845696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827773.112013:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827773.112018:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827773.112022:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972845696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827773.112029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827770/real 1787827773] req@ffff9c1a78860f00 x1874668972845824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827773.112043:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972845824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827773.112053:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827773.112057:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827773.112062:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972845824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827773.112070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827770/real 1787827773] req@ffff9c1a78861a40 x1874668972845952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827773.112084:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972845952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827773.112094:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827773.112098:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827773.112103:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972845952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827773.112111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827770/real 1787827773] req@ffff9c1a78860b40 x1874668972846080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827773.112123:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972846080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827773.112133:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827773.112137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827773.112141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972846080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827773.112150:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827770/real 1787827773] req@ffff9c1a78863840 x1874668972845440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827773.112163:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972845440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827773.112172:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827773.112176:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827773.112180:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972845440:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827773.112187:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827770/real 1787827773] req@ffff9c1a788612c0 x1874668972845568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827773.112199:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972845568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827773.112208:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827773.112211:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827773.112215:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972845568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827773.623427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827773.623434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827773.879525:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827774.647458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827774.647506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827774.903508:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827775.159489:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827775.159511:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827775.159515:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827775.159526:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4225 00000100:00080000:0.0:1787827775.159537:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827775.159545:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827775.159615:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972846336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827775.159649:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827775.159654:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827775.159656:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827775.159662:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4225 00000100:00080000:0.0:1787827775.159666:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827775.159670:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827775.159681:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827775.159700:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972846464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827775.159725:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827775.159729:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827775.159732:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827775.159738:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972846336:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827775.159741:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4225 00000100:00080000:0.0:1787827775.159750:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827775.159756:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827775.159762:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827775.159781:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972846592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827775.159798:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827775.159804:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827775.159808:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827775.159815:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4225 00000100:00080000:0.0:1787827775.159839:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827775.159857:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827775.159862:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827775.159874:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827775.159920:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827775/real 1787827775] req@ffff9c1a788621c0 x1874668972846336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827775.159925:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972846720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827775.159962:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827775.159967:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827775.159970:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827775.159978:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4225 00000100:00100000:1.0:1787827775.159982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972846336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827775.159988:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827775.160002:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827775.160009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827775.160012:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827775.160021:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827775.160029:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972846336:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827775.160038:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972846848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827775.160056:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827775.160060:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827775.160063:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827775.160068:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972846464:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827775.160075:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4225 00000100:00080000:0.0:1787827775.160085:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827775.160093:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827775.160101:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827775.160119:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972846976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827775.160135:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827775.160137:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827775.160156:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787827775.160160:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827775/real 1787827775] req@ffff9c1a78860780 x1874668972846464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827775.160161:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827775.160174:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4225 00000100:00080000:0.0:1787827775.160185:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827775.160194:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827775.160201:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972846464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827775.160204:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827775.160223:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827775.160231:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972847104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827775.160234:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827775.160246:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972846464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827775.160265:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972846592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827775.160301:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827775.160319:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827775/real 1787827775] req@ffff9c1a78861e00 x1874668972846592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827775.160391:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972846592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827775.160402:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827775.160406:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827775.160410:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972846592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827775.160418:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972846720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827775.160445:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827775.160460:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827775/real 1787827775] req@ffff9c1a78862580 x1874668972846720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827775.160476:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972846720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827775.160488:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827775.160492:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827775.160495:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972846720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827775.160503:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972846848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827775.160525:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827775.160547:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827775/real 1787827775] req@ffff9c1a78861680 x1874668972846848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827775.160570:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972846848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827775.160580:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827775.160583:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827775.160587:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972846848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827775.160613:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972846976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827775.160641:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827775.160656:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827775/real 1787827775] req@ffff9c1a788630c0 x1874668972846976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827775.160673:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972846976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827775.160685:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827775.160689:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827775.160693:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972846976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827775.160701:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972847104:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827775.160726:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827775.160743:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827775/real 1787827775] req@ffff9c1a78863c00 x1874668972847104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827775.160760:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972847104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827775.160780:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827775.160785:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827775.160789:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972847104:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827775.671460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827775.671460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827775.927504:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827776.695510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827776.695514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827776.951477:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827777.719422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827777.719429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827777.975530:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827778.743445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827778.743458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827778.999498:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827779.767434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827779.767434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827780.023650:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827780.279440:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827780.279455:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827780.279461:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827780.279476:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4230 00000100:00080000:3.0:1787827780.279491:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827780.279501:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827780.279553:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972847232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827780.279586:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827780.279590:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827780.279593:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827780.279598:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4230 00000100:00080000:3.0:1787827780.279603:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827780.279608:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827780.279619:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972847232:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787827780.279621:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827780.279639:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972847360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827780.279652:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827780.279657:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827780.279659:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827780.279664:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4230 00000100:00080000:3.0:1787827780.279670:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827780.279674:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827780.279679:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827780.279692:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972847488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827780.279695:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827780.279708:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827780.279713:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827780.279715:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827780.279721:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4230 00000100:00080000:3.0:1787827780.279725:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827780.279730:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827780.279735:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827780.279743:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827780/real 1787827780] req@ffff9c1a7895f840 x1874668972847232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827780.279750:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972847616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827780.279765:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827780.279766:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972847232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827780.279775:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827780.279780:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827780.279784:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827780.279787:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4230 00000100:00080000:1.0:1787827780.279791:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827780.279794:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827780.279796:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972847232:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827780.279802:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827780.279808:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827780.279820:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972847744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827780.279823:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972847360:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827780.279846:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827780.279847:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827780.279854:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827780.279860:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827780/real 1787827780] req@ffff9c1a7895f0c0 x1874668972847360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827780.279866:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827780.279871:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972847360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827780.279875:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4230 00000100:00080000:3.0:1787827780.279885:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827780.279886:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827780.279893:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827780.279896:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827780.279898:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972847360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827780.279907:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972847488:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787827780.279907:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827780.279926:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827780.279928:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972847872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827780.279936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827780/real 1787827780] req@ffff9c1a7895da40 x1874668972847488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827780.279949:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827780.279952:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972847488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827780.279957:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827780.279963:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827780.279964:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827780.279972:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827780.279974:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4230 00000100:00100000:1.0:1787827780.279978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972847488:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827780.279983:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827780.279985:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972847616:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827780.279992:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827780.279999:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827780.280003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827780.280012:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827780/real 1787827780] req@ffff9c1a7895e1c0 x1874668972847616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827780.280019:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972848000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827780.280025:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972847616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827780.280043:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827780.280047:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827780.280051:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972847616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827780.280062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972847744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827780.280076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827780.280086:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827780/real 1787827780] req@ffff9c1a7895de00 x1874668972847744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827780.280099:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972847744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827780.280108:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827780.280111:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827780.280114:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972847744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827780.280121:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972847872:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827780.280160:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827780.280170:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827780/real 1787827780] req@ffff9c1a7895c780 x1874668972847872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827780.280179:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972847872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827780.280186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827780.280189:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827780.280191:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972847872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827780.280195:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972848000:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827780.280208:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827780.280214:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827780/real 1787827780] req@ffff9c1a7895cb40 x1874668972848000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827780.280229:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972848000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827780.280250:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827780.280252:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827780.280254:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972848000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827780.791494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827780.791495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827781.047555:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827781.815468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827781.815468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827782.071486:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827782.839461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827782.839461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827783.095507:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827783.863442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827783.863442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827784.119463:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827784.887457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827784.887485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827785.143482:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827785.399459:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827785.399467:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827785.399471:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827785.399479:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4235 00000100:00080000:0.0:1787827785.399489:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827785.399495:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827785.399540:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972848128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827785.399567:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827785.399570:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827785.399571:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827785.399575:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4235 00000100:00080000:0.0:1787827785.399577:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827785.399580:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827785.399588:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827785.399599:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972848256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827785.399607:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827785.399609:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827785.399610:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827785.399613:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4235 00000100:00080000:0.0:1787827785.399615:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827785.399618:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827785.399623:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972848128:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787827785.399624:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827785.399638:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972848384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827785.399647:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827785.399650:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827785.399651:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827785.399654:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4235 00000100:00080000:0.0:1787827785.399657:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827785.399660:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827785.399663:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827785.399671:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972848512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827785.399677:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827785.399680:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827785.399681:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827785.399686:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4235 00000800:00000100:1.0:1787827785.399686:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827785.399692:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827785.399695:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827785.399699:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827785.399707:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972848640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827785.399714:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827785.399716:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827785.399718:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827785.399721:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4235 00000100:00080000:0.0:1787827785.399724:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827785.399727:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827785.399733:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827785.399733:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827785/real 1787827785] req@ffff9c1a78860000 x1874668972848128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827785.399744:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972848768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827785.399755:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827785.399758:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827785.399759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827785.399762:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4235 00000100:00080000:0.0:1787827785.399766:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827785.399767:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972848128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827785.399772:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827785.399777:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827785.399783:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972848896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827785.399801:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827785.399806:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827785.399810:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972848128:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827785.399817:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972848256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827785.399840:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827785.399850:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827785/real 1787827785] req@ffff9c1a78863c00 x1874668972848256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827785.399863:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972848256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827785.399869:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827785.399871:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827785.399874:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972848256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827785.399891:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972848384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827785.399903:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827785.399909:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827785/real 1787827785] req@ffff9c1a788630c0 x1874668972848384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827785.399917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972848384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827785.399922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827785.399924:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827785.399925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972848384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827785.399929:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972848512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827785.399939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827785.399945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827785/real 1787827785] req@ffff9c1a78861680 x1874668972848512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827785.399952:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972848512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827785.399963:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827785.399964:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827785.399966:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972848512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827785.399969:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972848640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827785.399979:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827785.399984:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827785/real 1787827785] req@ffff9c1a78862580 x1874668972848640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827785.399991:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972848640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827785.399996:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827785.399998:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827785.400000:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972848640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827785.400004:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972848768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827785.400014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827785.400020:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827785/real 1787827785] req@ffff9c1a78861e00 x1874668972848768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827785.400027:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972848768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827785.400032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827785.400034:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827785.400036:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972848768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827785.400039:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972848896:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827785.400048:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827785.400053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827785/real 1787827785] req@ffff9c1a78860780 x1874668972848896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827785.400059:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972848896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827785.400064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827785.400066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827785.400068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972848896:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827785.911449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827785.911452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827786.167558:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827786.935514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827786.935577:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827787.191451:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827787.959500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827787.959515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827788.215508:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827788.983418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827788.983478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827789.239452:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827790.007458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827790.007459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827790.263551:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827790.519452:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827790.519460:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827790.519464:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827790.519473:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4240 00000100:00080000:2.0:1787827790.519483:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827790.519489:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827790.519523:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972849024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827790.519549:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827790.519553:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827790.519556:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827790.519560:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4240 00000100:00080000:2.0:1787827790.519565:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827790.519571:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827790.519582:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827790.519591:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972849024:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827790.519596:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972849152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827790.519605:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827790.519608:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827790.519609:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827790.519612:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4240 00000100:00080000:2.0:1787827790.519614:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827790.519616:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827790.519620:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827790.519628:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972849280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827790.519636:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827790.519639:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827790.519640:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827790.519642:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4240 00000100:00080000:2.0:1787827790.519645:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827790.519647:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827790.519650:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827790.519657:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972849408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827790.519669:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827790.519672:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827790.519673:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827790.519676:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827790.519680:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4240 00000100:00080000:2.0:1787827790.519683:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827790.519685:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827790.519688:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827790.519701:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972849536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827790.519708:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827790.519710:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827790.519711:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827790.519717:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4240 00000100:00100000:1.0:1787827790.519718:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827790/real 1787827790] req@ffff9c1a49d8f0c0 x1874668972849024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827790.519720:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827790.519726:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827790.519732:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827790.519742:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972849664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827790.519749:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972849024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827790.519752:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827790.519757:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827790.519759:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827790.519767:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4240 00000100:00080000:2.0:1787827790.519774:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827790.519778:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827790.519779:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827790.519785:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827790.519789:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827790.519795:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972849792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827790.519796:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972849024:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827790.519826:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972849152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827790.519849:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827790.519862:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827790/real 1787827790] req@ffff9c1a49d8c780 x1874668972849152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827790.519877:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972849152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827790.519887:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827790.519890:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827790.519894:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972849152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827790.519900:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972849280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827790.519916:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827790.519923:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827790/real 1787827790] req@ffff9c1a49d8de00 x1874668972849280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827790.519931:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972849280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827790.519938:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827790.519941:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827790.519945:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972849280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827790.519951:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972849408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827790.519980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827790.519992:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827790/real 1787827790] req@ffff9c1a49d8c000 x1874668972849408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827790.520004:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972849408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827790.520013:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827790.520016:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827790.520019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972849408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827790.520037:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972849536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827790.520059:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827790.520069:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827790/real 1787827790] req@ffff9c1a49d8da40 x1874668972849536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827790.520077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972849536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827790.520083:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827790.520085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827790.520087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972849536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827790.520091:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972849664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827790.520149:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827790.520162:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827790/real 1787827790] req@ffff9c1a49d8cf00 x1874668972849664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827790.520172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972849664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827790.520186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827790.520189:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827790.520191:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972849664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827790.520196:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972849792:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827790.520211:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827790.520252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827790/real 1787827790] req@ffff9c1a49d8cb40 x1874668972849792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827790.520262:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972849792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827790.520269:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827790.520272:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827790.520274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972849792:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827791.031420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827791.031425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827791.287526:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827792.055430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827792.055433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827792.311539:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827793.079418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827793.079422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827793.335492:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827794.103443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827794.103449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827794.359468:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827795.127431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827795.127491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827795.383583:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827795.639514:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827795.639522:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827795.639542:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827795.639551:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4245 00000100:00080000:0.0:1787827795.639561:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827795.639567:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827795.639624:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972849920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827795.639673:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827795.639678:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827795.639680:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827795.639685:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4245 00000100:00080000:0.0:1787827795.639689:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827795.639693:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827795.639704:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827795.639723:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972850048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827795.639747:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972849920:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827795.639757:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827795.639762:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827795.639764:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827795.639769:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4245 00000100:00080000:0.0:1787827795.639773:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827795.639777:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827795.639783:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827795.639793:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972850176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827795.639820:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827795.639824:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827795.639825:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827795.639829:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4245 00000100:00080000:0.0:1787827795.639834:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827795.639840:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827795.639844:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827795.639851:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827795.639863:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972850304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827795.639872:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827795.639875:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827795.639877:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827795.639897:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4245 00000100:00080000:0.0:1787827795.639900:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827795.639905:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827795.639906:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827795/real 1787827795] req@ffff9c1a788621c0 x1874668972849920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827795.639909:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827795.639922:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972850432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827795.639929:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827795.639930:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972849920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827795.639936:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827795.639940:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827795.639962:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4245 00000100:00080000:0.0:1787827795.639968:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827795.639971:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827795.639974:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827795.639988:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827795.639989:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972850560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827795.639996:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827795.640005:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972849920:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827795.640007:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827795.640013:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827795.640014:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827795.640018:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4245 00000100:00080000:0.0:1787827795.640021:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827795.640024:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827795.640026:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827795.640036:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972850048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827795.640038:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972850688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827795.640068:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827795.640081:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827795/real 1787827795] req@ffff9c1a788612c0 x1874668972850048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827795.640091:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972850048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827795.640098:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827795.640101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827795.640106:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972850048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827795.640113:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972850176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827795.640136:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827795.640149:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827795/real 1787827795] req@ffff9c1a78863840 x1874668972850176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827795.640164:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972850176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827795.640172:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827795.640175:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827795.640177:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972850176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827795.640198:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972850304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827795.640216:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827795.640226:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827795/real 1787827795] req@ffff9c1a78860b40 x1874668972850304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827795.640235:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972850304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827795.640241:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827795.640243:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827795.640246:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972850304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827795.640250:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972850432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827795.640262:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827795.640268:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827795/real 1787827795] req@ffff9c1a78861a40 x1874668972850432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827795.640276:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972850432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827795.640290:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827795.640293:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827795.640295:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972850432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827795.640300:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972850560:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827795.640313:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827795.640320:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827795/real 1787827795] req@ffff9c1a78860f00 x1874668972850560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827795.640330:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972850560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827795.640336:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827795.640338:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827795.640340:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972850560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827795.640387:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972850688:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827795.640402:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827795.640411:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827795/real 1787827795] req@ffff9c1a78862d00 x1874668972850688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827795.640419:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972850688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827795.640426:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827795.640428:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827795.640430:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972850688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827796.151528:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827796.151528:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827796.407556:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827797.175413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827797.175414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827797.431486:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827798.199448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827798.199449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827798.455510:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827799.223438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827799.223440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827799.479483:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827800.247491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827800.247495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827800.503583:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827800.759596:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827800.759606:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827800.759611:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827800.759623:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4251 00000100:00080000:2.0:1787827800.759638:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827800.759648:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827800.759698:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972850816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827800.759743:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827800.759749:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827800.759753:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827800.759759:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4251 00000100:00080000:2.0:1787827800.759768:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827800.759773:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827800.759785:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827800.759802:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972850944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827800.759817:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972850816:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827800.759829:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827800.759837:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827800.759857:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827800.759867:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4251 00000100:00080000:2.0:1787827800.759874:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827800.759882:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827800.759892:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827800.759914:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972851072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827800.759930:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827800.759941:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827800.759946:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827800.759950:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827800.759957:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4251 00000100:00080000:2.0:1787827800.759965:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827800.759971:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827800.759979:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827800.759994:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827800/real 1787827800] req@ffff9c1a49d8d680 x1874668972850816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827800.759996:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972851200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827800.760091:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827800.760102:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827800.760106:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827800.760115:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4251 00000100:00100000:1.0:1787827800.760117:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972850816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827800.760121:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827800.760138:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827800.760159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827800.760184:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:2.0:1787827800.760186:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827800.760194:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972850816:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827800.760208:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972851328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827800.760219:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827800.760223:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827800.760243:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827800.760250:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972850944:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787827800.760253:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4251 00000100:00080000:2.0:1787827800.760268:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827800.760276:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827800.760285:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827800.760311:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827800.760313:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972851456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827800.760352:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827800/real 1787827800] req@ffff9c1a49d8e1c0 x1874668972850944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827800.760354:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827800.760365:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827800.760370:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827800.760453:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4251 00000100:00100000:1.0:1787827800.760461:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972850944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827800.760468:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827800.760507:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827800.760511:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827800.760518:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000008:00100000:2.0:1787827800.760519:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827800.760529:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972850944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827800.760537:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972851072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827800.760545:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972851584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827800.760571:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827800.760604:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827800/real 1787827800] req@ffff9c1a49d8c3c0 x1874668972851072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827800.760617:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972851072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827800.760628:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827800.760631:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827800.760633:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972851072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827800.760638:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972851200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827800.760654:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827800.760662:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827800/real 1787827800] req@ffff9c1a49d8e580 x1874668972851200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827800.760670:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972851200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827800.760676:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827800.760678:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827800.760680:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972851200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827800.760684:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972851328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827800.760695:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827800.760701:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827800/real 1787827800] req@ffff9c1a49d8fc00 x1874668972851328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827800.760708:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972851328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827800.760713:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827800.760715:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827800.760717:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972851328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827800.760734:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972851456:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827800.760746:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827800.760752:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827800/real 1787827800] req@ffff9c1a49d8ed00 x1874668972851456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827800.760760:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972851456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827800.760765:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827800.760767:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827800.760769:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972851456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827800.760773:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972851584:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827800.760782:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827800.760788:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827800/real 1787827800] req@ffff9c1a49d8d680 x1874668972851584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827800.760796:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972851584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827800.760802:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827800.760803:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827800.760805:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972851584:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827801.271490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827801.271515:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827801.527477:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827802.295462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827802.295470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827802.551541:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827803.319462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827803.319462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827803.575510:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827804.343421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827804.343428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827804.599530:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827805.367484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827805.367490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827805.623509:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827805.879566:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827805.879583:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827805.879589:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827805.879605:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4256 00000100:00080000:0.0:1787827805.879633:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827805.879645:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827805.879705:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972851712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827805.879778:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827805.879784:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827805.879787:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827805.879794:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4256 00000100:00080000:0.0:1787827805.879801:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827805.879808:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827805.879823:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827805.879845:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972851840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827805.879869:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972851712:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827805.879878:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827805.879885:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827805.879888:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827805.879895:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4256 00000100:00080000:0.0:1787827805.879901:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827805.879905:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827805.879911:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827805.879930:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972851968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827805.879946:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827805.879951:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827805.879954:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827805.879966:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4256 00000800:00000100:1.0:1787827805.879967:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827805.879977:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827805.879985:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827805.880002:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827805.880019:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972852096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827805.880041:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827805/real 1787827805] req@ffff9c1a78862940 x1874668972851712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827805.880044:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827805.880060:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827805.880067:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827805.880082:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4256 00000100:00080000:0.0:1787827805.880089:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827805.880096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972851712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827805.880100:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827805.880161:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827805.880180:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827805.880194:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972852224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827805.880195:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827805.880208:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972851712:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827805.880235:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827805.880240:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827805.880243:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827805.880250:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4256 00000100:00080000:0.0:1787827805.880256:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827805.880265:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972851840:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827805.880266:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827805.880283:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827805.880303:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972852352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827805.880318:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827805.880327:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827805.880328:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827805.880338:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827805.880402:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4256 00000100:00080000:0.0:1787827805.880410:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827805.880415:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827805.880423:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827805.880430:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827805/real 1787827805] req@ffff9c1a78862d00 x1874668972851840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827805.880444:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972852480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827805.880473:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972851840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827805.880502:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827805.880509:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827805.880516:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972851840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827805.880530:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972851968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827805.880571:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827805.880588:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827805/real 1787827805] req@ffff9c1a78860f00 x1874668972851968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827805.880622:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972851968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827805.880632:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827805.880636:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827805.880640:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972851968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827805.880647:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972852096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827805.880667:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827805.880678:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827805/real 1787827805] req@ffff9c1a78861a40 x1874668972852096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827805.880691:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972852096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827805.880700:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827805.880703:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827805.880707:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972852096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827805.880713:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972852224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827805.880730:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827805.880749:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827805/real 1787827805] req@ffff9c1a78860b40 x1874668972852224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827805.880762:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972852224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827805.880779:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827805.880782:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827805.880786:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972852224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827805.880807:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972852352:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827805.880846:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827805.880859:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827805/real 1787827805] req@ffff9c1a78863840 x1874668972852352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827805.880883:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972852352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827805.880894:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827805.880897:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827805.880901:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972852352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827805.880908:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972852480:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827805.880925:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827805.880937:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827805/real 1787827805] req@ffff9c1a788612c0 x1874668972852480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827805.880960:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972852480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827805.880975:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827805.880982:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827805.880989:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972852480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827806.391525:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827806.391544:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827806.647589:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827807.415475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827807.415484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827807.671612:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827808.439466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827808.439490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827808.695573:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827809.463567:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827809.463581:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827809.719596:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827810.487422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827810.487422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827810.743710:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827810.999417:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827810.999425:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827810.999429:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827810.999437:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4261 00000100:00080000:0.0:1787827810.999446:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827810.999453:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827810.999485:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972852608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827810.999509:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827810.999512:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827810.999514:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827810.999519:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4261 00000100:00080000:0.0:1787827810.999522:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827810.999526:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827810.999536:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827810.999546:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972852608:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827810.999554:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972852736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827810.999569:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827810.999575:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827810.999578:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827810.999584:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4261 00000100:00080000:0.0:1787827810.999589:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827810.999594:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827810.999599:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827810.999611:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972852864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827810.999620:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827810.999623:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827810.999625:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827810.999628:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4261 00000100:00080000:0.0:1787827810.999630:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827810.999634:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827810.999637:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827810.999643:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827810.999648:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972852992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827810.999655:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827810.999657:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827810.999659:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827810.999662:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4261 00000100:00080000:0.0:1787827810.999665:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827810.999668:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827810.999672:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827810.999681:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972853120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827810.999688:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827810.999691:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827810/real 1787827810] req@ffff9c1a788621c0 x1874668972852608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827810.999692:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827810.999696:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827810.999704:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4261 00000100:00080000:0.0:1787827810.999711:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827810.999714:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827810.999718:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827810.999719:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972852608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827810.999729:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972853248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827810.999732:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827810.999743:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827810.999745:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827810.999749:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827810.999751:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827810.999758:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4261 00000100:00100000:2.0:1787827810.999758:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972852608:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827810.999764:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827810.999769:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827810.999772:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827810.999779:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972853376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827810.999788:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972852736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827810.999810:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827810.999823:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827810/real 1787827810] req@ffff9c1a78860780 x1874668972852736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827810.999833:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972852736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827810.999840:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827810.999843:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827810.999846:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972852736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827810.999850:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972852864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827810.999864:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827810.999872:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827810/real 1787827810] req@ffff9c1a78861e00 x1874668972852864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827810.999880:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972852864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827810.999887:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827810.999889:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827810.999892:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972852864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827810.999896:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972852992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827810.999910:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827810.999918:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827810/real 1787827810] req@ffff9c1a78862580 x1874668972852992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827810.999929:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972852992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827810.999936:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827810.999938:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827810.999941:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972852992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827810.999946:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972853120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827810.999961:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827810.999976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827810/real 1787827810] req@ffff9c1a78861680 x1874668972853120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827810.999986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972853120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827810.999993:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827810.999996:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827811.000000:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972853120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827811.000004:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972853248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827811.000016:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827811.000023:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827810/real 1787827811] req@ffff9c1a788630c0 x1874668972853248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827811.000031:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972853248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827811.000041:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827811.000043:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827811.000046:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972853248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827811.000059:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972853376:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827811.000070:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827811.000092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827810/real 1787827811] req@ffff9c1a78863c00 x1874668972853376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827811.000101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972853376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827811.000107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827811.000109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827811.000111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972853376:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827811.511484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827811.511485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827811.767609:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827812.535427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827812.535429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827812.791453:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827813.559403:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827813.559456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827813.815607:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827814.583450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827814.583520:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827814.839600:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827815.607461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827815.607461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827815.863641:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827816.119493:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827816.119506:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827816.119512:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827816.119526:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4266 00000100:00080000:0.0:1787827816.119539:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827816.119549:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827816.119601:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972853504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827816.119651:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827816.119656:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827816.119658:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827816.119664:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4266 00000100:00080000:0.0:1787827816.119668:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827816.119672:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827816.119685:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827816.119700:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972853632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827816.119713:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827816.119717:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827816.119719:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827816.119723:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4266 00000100:00080000:0.0:1787827816.119727:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827816.119731:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827816.119736:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827816.119749:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972853760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827816.119766:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827816.119771:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827816.119774:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827816.119783:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4266 00000100:00100000:2.0:1787827816.119785:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972853504:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827816.119787:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827816.119799:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827816.119807:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827816.119823:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972853888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827816.119852:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827816.119857:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827816.119860:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827816.119866:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4266 00000100:00080000:0.0:1787827816.119872:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827816.119877:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787827816.119885:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787827816.119901:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827816.119925:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972854016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827816.119938:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827816.119942:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827816.119945:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827816.119952:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4266 00000100:00100000:2.0:1787827816.119960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827816/real 1787827816] req@ffff9c1a78860000 x1874668972853504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827816.119963:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827816.119978:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827816.119992:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827816.120019:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972854144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827816.120029:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972853504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827816.120062:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827816.120090:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827816.120093:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827816.120097:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827816.120103:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4266 00000100:00080000:2.0:1787827816.120112:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827816.120114:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827816.120124:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972853504:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827816.120127:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827816.120143:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827816.120151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972853632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827816.120166:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972854272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827816.120198:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827816.120221:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827816/real 1787827816] req@ffff9c1a78863c00 x1874668972853632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827816.120238:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972853632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827816.120249:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827816.120253:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827816.120257:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972853632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827816.120264:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972853760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827816.120294:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827816.120305:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827816/real 1787827816] req@ffff9c1a788630c0 x1874668972853760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827816.120319:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972853760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827816.120337:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827816.120340:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827816.120386:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972853760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827816.120433:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972853888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827816.120476:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827816.120501:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827816/real 1787827816] req@ffff9c1a78861680 x1874668972853888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827816.120526:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972853888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827816.120543:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827816.120550:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827816.120556:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972853888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827816.120569:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972854016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827816.120609:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827816.120635:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827816/real 1787827816] req@ffff9c1a78862580 x1874668972854016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827816.120660:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972854016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827816.120677:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827816.120683:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827816.120691:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972854016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827816.120703:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972854144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827816.120743:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827816.120762:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827816/real 1787827816] req@ffff9c1a78861e00 x1874668972854144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827816.120777:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972854144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827816.120788:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827816.120791:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827816.120795:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972854144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827816.120803:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972854272:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827816.120825:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827816.120836:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827816/real 1787827816] req@ffff9c1a78860780 x1874668972854272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827816.120850:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972854272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827816.120859:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827816.120863:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827816.120869:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972854272:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827816.631412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827816.631425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827816.887645:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827817.655437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827817.655441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827817.911643:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827818.679493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827818.679504:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827818.935514:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827819.703433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827819.703556:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827819.959476:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827820.727492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827820.727499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827820.983650:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827821.239470:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827821.239484:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827821.239489:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827821.239513:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4271 00000100:00080000:0.0:1787827821.239527:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827821.239536:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827821.239598:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972854400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827821.239647:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827821.239671:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827821.239675:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827821.239684:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4271 00000100:00080000:0.0:1787827821.239692:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827821.239701:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827821.239718:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827821.239746:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972854528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827821.239779:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972854400:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827821.239787:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827821.239798:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827821.239802:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827821.239811:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4271 00000100:00080000:0.0:1787827821.239819:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827821.239828:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827821.239850:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827821.239876:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972854656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827821.239891:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827821.239900:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1787827821.239901:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827821.239921:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827821.239930:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4271 00000100:00080000:0.0:1787827821.239939:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827821.239945:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827821.239953:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827821.239974:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972854784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827821.239983:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827821/real 1787827821] req@ffff9c1a788621c0 x1874668972854400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827821.239997:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827821.240033:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827821.240039:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827821.240051:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4271 00000100:00080000:0.0:1787827821.240059:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827821.240066:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972854400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827821.240069:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787827821.240089:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827821.240092:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827821.240101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827821.240116:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972854400:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827821.240137:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972854912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827821.240154:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972854528:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827821.240156:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827821.240170:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827821.240173:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827821.240180:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4271 00000100:00080000:0.0:1787827821.240188:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827821.240193:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827821.240201:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827821.240204:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827821.240227:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972855040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827821.240236:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827821/real 1787827821] req@ffff9c1a788612c0 x1874668972854528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827821.240256:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827821.240261:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827821.240264:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827821.240271:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4271 00000100:00080000:0.0:1787827821.240282:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827821.240289:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827821.240295:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827821.240296:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972854528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827821.240315:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972855168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827821.240317:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827821.240332:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827821.240412:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972854528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827821.240431:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972854656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827821.240485:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827821.240519:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827821/real 1787827821] req@ffff9c1a78863840 x1874668972854656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827821.240541:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972854656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827821.240551:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827821.240556:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827821.240560:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972854656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827821.240567:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972854784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827821.240600:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827821.240613:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827821/real 1787827821] req@ffff9c1a78860b40 x1874668972854784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827821.240628:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972854784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827821.240638:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827821.240641:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827821.240645:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972854784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827821.240666:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972854912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827821.240685:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827821.240696:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827821/real 1787827821] req@ffff9c1a78861a40 x1874668972854912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827821.240719:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972854912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827821.240729:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827821.240732:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827821.240736:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972854912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827821.240742:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972855040:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827821.240769:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827821.240780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827821/real 1787827821] req@ffff9c1a78860f00 x1874668972855040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827821.240794:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972855040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827821.240803:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827821.240807:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827821.240810:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972855040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827821.240817:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972855168:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827821.240834:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827821.240847:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827821/real 1787827821] req@ffff9c1a78862d00 x1874668972855168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827821.240860:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972855168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827821.240878:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827821.240882:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827821.240886:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972855168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827821.751451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827821.751503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827822.007577:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827822.775506:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827822.775576:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827823.031610:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827823.799525:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827823.799545:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827824.055625:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827824.823482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827824.823497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827825.079582:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827825.847431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827825.847431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827826.103532:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827826.359510:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827826.359521:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827826.359527:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827826.359560:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4276 00000100:00080000:3.0:1787827826.359577:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827826.359590:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827826.359649:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972855296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827826.359693:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827826.359698:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827826.359702:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827826.359707:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4276 00000100:00080000:3.0:1787827826.359713:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827826.359719:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827826.359748:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827826.359765:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972855424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827826.359776:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827826.359780:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827826.359783:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827826.359787:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4276 00000100:00080000:3.0:1787827826.359791:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827826.359797:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827826.359802:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827826.359813:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972855552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827826.359826:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827826.359829:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972855296:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827826.359834:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827826.359842:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827826.359850:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4276 00000100:00080000:3.0:1787827826.359855:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827826.359877:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827826.359884:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827826.359905:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972855680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827826.359920:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827826.359928:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1787827826.359929:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827826.359934:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827826.359941:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4276 00000100:00080000:3.0:1787827826.359946:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827826.359950:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827826.359963:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827826.360006:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972855808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827826.360017:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827826/real 1787827826] req@ffff9c1a7895d2c0 x1874668972855296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827826.360020:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827826.360029:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827826.360032:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827826.360042:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4276 00000100:00080000:3.0:1787827826.360050:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827826.360056:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827826.360061:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827826.360069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972855296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827826.360073:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972855936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827826.360114:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827826.360119:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827826.360119:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827826.360121:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827826.360134:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827826.360135:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4276 00000100:00080000:3.0:1787827826.360142:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827826.360148:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972855296:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827826.360151:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827826.360160:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827826.360165:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972855424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827826.360177:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972856064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827826.360203:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827826.360226:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827826/real 1787827826] req@ffff9c1a7895cb40 x1874668972855424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827826.360247:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972855424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827826.360260:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827826.360264:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827826.360267:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972855424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827826.360292:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972855552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827826.360314:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827826.360324:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827826/real 1787827826] req@ffff9c1a7895c780 x1874668972855552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827826.360334:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972855552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827826.360342:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827826.360377:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827826.360380:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972855552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827826.360386:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972855680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827826.360404:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827826.360418:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827826/real 1787827826] req@ffff9c1a7895de00 x1874668972855680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827826.360436:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972855680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827826.360459:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827826.360464:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827826.360469:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972855680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827826.360478:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972855808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827826.360505:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827826.360517:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827826/real 1787827826] req@ffff9c1a7895e1c0 x1874668972855808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827826.360528:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972855808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827826.360535:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827826.360538:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827826.360541:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972855808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827826.360546:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972855936:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827826.360562:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827826.360570:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827826/real 1787827826] req@ffff9c1a7895da40 x1874668972855936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827826.360580:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972855936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827826.360587:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827826.360589:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827826.360592:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972855936:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827826.360596:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972856064:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827826.360609:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827826.360617:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827826/real 1787827826] req@ffff9c1a7895f0c0 x1874668972856064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827826.360627:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972856064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827826.360634:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827826.360637:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827826.360639:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972856064:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827826.871462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827826.871465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827827.127522:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827827.895458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827827.895467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827828.151608:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827828.919476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827828.919489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827829.175632:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827829.943450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827829.943492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827830.199472:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827830.967424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827830.967436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827831.223595:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827831.479525:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827831.479542:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827831.479549:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827831.479564:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4281 00000100:00080000:3.0:1787827831.479578:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827831.479586:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827831.479644:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972856192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827831.479700:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827831.479705:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827831.479708:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827831.479713:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4281 00000100:00080000:3.0:1787827831.479717:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827831.479722:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827831.479734:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827831.479748:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972856320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827831.479763:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827831.479768:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827831.479771:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827831.479776:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4281 00000100:00080000:3.0:1787827831.479780:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827831.479784:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827831.479791:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827831.479820:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972856192:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827831.479821:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972856448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827831.479849:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827831.479853:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827831.479855:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827831.479861:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4281 00000100:00080000:3.0:1787827831.479865:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827831.479870:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827831.479876:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827831.479891:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972856576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827831.479903:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827831.479907:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827831.479909:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827831.479919:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827831.479920:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4281 00000100:00080000:3.0:1787827831.479952:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827831.479970:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827831.479978:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827831.479996:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972856704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827831.480010:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827831.480014:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827831.480021:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827831.480023:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827831/real 1787827831] req@ffff9c1a7895f840 x1874668972856192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827831.480031:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4281 00000100:00080000:3.0:1787827831.480042:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827831.480054:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827831.480063:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827831.480071:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972856192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827831.480082:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972856832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827831.480106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827831.480110:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827831.480118:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827831.480120:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827831.480127:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827831.480132:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972856192:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827831.480138:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4281 00000100:00080000:3.0:1787827831.480147:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827831.480155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972856320:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827831.480164:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827831.480178:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827831.480198:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972856960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827831.480212:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827831.480243:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827831/real 1787827831] req@ffff9c1a7895c000 x1874668972856320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827831.480269:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972856320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827831.480294:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827831.480301:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827831.480308:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972856320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827831.480428:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972856448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827831.480480:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827831.480506:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827831/real 1787827831] req@ffff9c1a7895f480 x1874668972856448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827831.480543:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972856448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827831.480561:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827831.480567:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827831.480574:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972856448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827831.480587:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972856576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827831.480627:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827831.480650:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827831/real 1787827831] req@ffff9c1a7895cf00 x1874668972856576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827831.480690:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972856576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827831.480708:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827831.480715:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827831.480722:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972856576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827831.480731:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972856704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827831.480761:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827831.480778:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827831/real 1787827831] req@ffff9c1a7895c3c0 x1874668972856704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827831.480804:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972856704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827831.480832:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827831.480838:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827831.480844:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972856704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827831.480857:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972856832:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827831.480909:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827831.480960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827831/real 1787827831] req@ffff9c1a7895fc00 x1874668972856832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827831.480999:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972856832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827831.481040:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827831.481046:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827831.481053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972856832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827831.481062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972856960:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827831.481097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827831.481129:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827831/real 1787827831] req@ffff9c1a7895e580 x1874668972856960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827831.481151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972856960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827831.481167:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827831.481173:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827831.481179:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972856960:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827831.991509:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827831.991513:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827832.247595:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827833.015421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827833.015421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827833.271528:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827834.039421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827834.039422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827834.295461:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827835.063480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827835.063532:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827835.319504:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827836.087388:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827836.087417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827836.343472:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827836.599512:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827836.599520:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827836.599524:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827836.599533:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4286 00000100:00080000:0.0:1787827836.599542:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827836.599548:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827836.599584:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972857088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827836.599609:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827836.599613:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827836.599615:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827836.599619:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4286 00000100:00080000:0.0:1787827836.599624:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827836.599628:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827836.599639:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827836.599670:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972857216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827836.599683:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972857088:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827836.599684:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827836.599692:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827836.599694:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827836.599698:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4286 00000100:00080000:0.0:1787827836.599702:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827836.599707:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827836.599712:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827836.599725:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972857344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827836.599737:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827836.599740:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827836.599742:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827836.599744:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4286 00000100:00080000:0.0:1787827836.599748:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827836.599751:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827836.599754:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827836.599764:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972857472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827836.599771:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827836.599773:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827836.599774:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827836.599794:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4286 00000100:00080000:0.0:1787827836.599797:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827836.599800:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972857216:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827836.599803:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827836.599810:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827836.599821:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972857600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827836.599831:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827836.599834:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827836.599836:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972857344:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827836.599838:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827836.599847:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4286 00000100:00080000:0.0:1787827836.599853:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827836.599858:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827836.599865:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827836.599872:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972857472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827836.599892:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972857728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827836.599936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972857600:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827836.599940:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827836.599945:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827836.599947:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827836.599953:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4286 00000100:00080000:0.0:1787827836.599958:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827836.599962:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827836.599968:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827836.599973:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972857728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827836.599983:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972857856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827836.600007:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972857856:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827837.111426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827837.111441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827837.367536:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827838.135459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827838.135470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827838.391578:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827839.159470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827839.159470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827839.415492:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827839.671766:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787827839.671821:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827839.671844:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827839.671917:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827839.671943:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827839.671964:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827839.671985:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827839.672001:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787827839.672015:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787827839.672126:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827836/real 1787827839] req@ffff9c1a788612c0 x1874668972857856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827839.672152:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972857856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827839.672166:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827839.672172:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827839.672177:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972857856:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827839.672186:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827836/real 1787827839] req@ffff9c1a78863840 x1874668972857728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827839.672201:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972857728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827839.672211:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827839.672215:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827839.672219:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972857728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827839.672226:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827836/real 1787827839] req@ffff9c1a78860b40 x1874668972857600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827839.672240:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972857600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827839.672250:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827839.672255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827839.672259:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972857600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827839.672272:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827836/real 1787827839] req@ffff9c1a78862d00 x1874668972857216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827839.672287:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972857216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827839.672296:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827839.672300:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827839.672303:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972857216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827839.672310:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827836/real 1787827839] req@ffff9c1a78860f00 x1874668972857344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827839.672324:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972857344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827839.672334:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827839.672337:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827839.672341:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972857344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827839.673020:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827836/real 1787827839] req@ffff9c1a78861a40 x1874668972857472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827839.673067:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972857472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827839.673087:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827839.673093:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827839.673097:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972857472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827839.673106:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827836/real 1787827839] req@ffff9c1a78862940 x1874668972857088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827839.673124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972857088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827839.673135:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827839.673141:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827839.673146:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972857088:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827840.183434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827840.183446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827840.439599:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827841.207446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827841.207446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827841.463472:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827841.719539:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827841.719548:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827841.719552:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827841.719562:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4292 00000100:00080000:1.0:1787827841.719572:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827841.719579:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827841.719622:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972857984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827841.719655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827841.719663:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827841.719665:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827841.719669:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4292 00000100:00080000:1.0:1787827841.719672:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827841.719675:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827841.719684:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827841.719694:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972858112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827841.719704:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827841.719709:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827841.719711:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827841.719716:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4292 00000100:00080000:1.0:1787827841.719721:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827841.719727:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827841.719735:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827841.719754:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972857984:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827841.719755:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972858240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827841.719776:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827841.719781:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827841.719784:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827841.719790:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4292 00000100:00080000:1.0:1787827841.719796:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827841.719803:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827841.719810:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827841.719828:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972858368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827841.719834:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827841.719849:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827841.719854:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827841.719857:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827841.719877:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4292 00000100:00080000:1.0:1787827841.719883:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827841.719889:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827841.719896:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827841.719905:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827841/real 1787827841] req@ffff9c1a78a79e00 x1874668972857984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827841.719915:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972858496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827841.719941:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827841.719946:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787827841.719949:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972857984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827841.719950:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827841.719964:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4292 00000100:00080000:1.0:1787827841.719979:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827841.719984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827841.719989:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787827841.719997:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:1.0:1787827841.720002:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827841.720006:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972857984:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827841.720024:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972858624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827841.720025:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972858112:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827841.720056:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827841.720061:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827841.720064:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827841.720073:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4292 00000800:00000100:3.0:1787827841.720074:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827841.720082:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827841.720090:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827841.720097:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827841.720102:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827841/real 1787827841] req@ffff9c1a78a7b0c0 x1874668972858112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827841.720119:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972858752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827841.720124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972858112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827841.720158:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827841.720164:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827841.720169:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972858112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827841.720202:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972858240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827841.720241:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827841.720265:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827841/real 1787827841] req@ffff9c1a78a7bc00 x1874668972858240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827841.720283:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972858240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827841.720291:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827841.720294:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827841.720296:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972858240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827841.720305:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972858368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827841.720333:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827841.720695:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827841/real 1787827841] req@ffff9c1a78a78780 x1874668972858368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827841.720717:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972858368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827841.720728:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827841.720733:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827841.720737:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972858368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827841.720746:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972858496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827841.720779:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827841.720796:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827841/real 1787827841] req@ffff9c1a78a7a1c0 x1874668972858496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827841.720812:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972858496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827841.720821:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827841.720825:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827841.720829:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972858496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827841.720837:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972858624:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827841.720884:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827841.720902:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827841/real 1787827841] req@ffff9c1a78a783c0 x1874668972858624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827841.720919:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972858624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827841.720929:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827841.720933:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827841.720936:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972858624:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827841.720944:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972858752:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827841.720969:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827841.720984:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827841/real 1787827841] req@ffff9c1a78a7a940 x1874668972858752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827841.721001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972858752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827841.721011:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827841.721015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827841.721019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972858752:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827842.231450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827842.231454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827842.487474:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827843.255482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827843.255513:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827843.511592:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827844.279451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827844.279451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827844.535487:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827845.303450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827845.303466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827845.559472:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827846.327480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827846.327592:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827846.583496:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827846.839491:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827846.839501:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827846.839508:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827846.839538:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4297 00000100:00080000:2.0:1787827846.839552:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827846.839562:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827846.839608:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972858880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827846.839648:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827846.839653:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827846.839673:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827846.839679:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4297 00000100:00080000:2.0:1787827846.839685:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827846.839691:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827846.839703:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827846.839724:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972859008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827846.839756:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827846.839767:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827846.839770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972858880:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827846.839791:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827846.839801:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4297 00000100:00080000:2.0:1787827846.839808:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827846.839819:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827846.839877:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827846.839907:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972859136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827846.839953:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827846.839958:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1787827846.839969:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787827846.839971:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827846.839994:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4297 00000100:00080000:2.0:1787827846.840006:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827846.840013:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827846.840021:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827846.840040:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972859264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827846.840052:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827846.840056:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827846.840058:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827846.840062:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4297 00000100:00080000:2.0:1787827846.840065:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827846.840068:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827846.840074:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827846.840086:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972859392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827846.840096:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827846/real 1787827846] req@ffff9c1a49d8cb40 x1874668972858880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827846.840129:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827846.840136:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827846.840140:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827846.840149:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4297 00000100:00080000:2.0:1787827846.840157:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827846.840165:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827846.840175:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827846.840215:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972859520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827846.840241:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827846.840254:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827846.840258:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827846.840263:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4297 00000100:00080000:2.0:1787827846.840268:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827846.840272:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827846.840278:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827846.840291:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972859648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827846.840545:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972858880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827846.840557:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827846.840578:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827846.840583:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972858880:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827846.840636:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972859008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827846.840671:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827846.840687:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827846/real 1787827846] req@ffff9c1a49d8cf00 x1874668972859008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827846.840699:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972859008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827846.840708:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827846.840711:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827846.840715:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972859008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827846.840720:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972859136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827846.840737:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827846.840745:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827846/real 1787827846] req@ffff9c1a49d8da40 x1874668972859136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827846.840755:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972859136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827846.840763:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827846.840765:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827846.840768:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972859136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827846.840774:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972859264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827846.840789:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827846.840797:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827846/real 1787827846] req@ffff9c1a49d8c000 x1874668972859264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827846.840814:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972859264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827846.840822:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827846.840862:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827846.840865:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972859264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827846.840870:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972859392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827846.840887:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827846.840895:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827846/real 1787827846] req@ffff9c1a49d8de00 x1874668972859392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827846.840903:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972859392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827846.840909:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827846.840928:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827846.840931:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972859392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827846.840937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972859520:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827846.840956:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827846.840963:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827846/real 1787827846] req@ffff9c1a49d8c780 x1874668972859520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827846.840973:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972859520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827846.840987:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827846.840990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827846.840992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972859520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827846.840998:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972859648:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827846.841011:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827846.841018:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827846/real 1787827846] req@ffff9c1a49d8f0c0 x1874668972859648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827846.841028:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972859648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827846.841035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827846.841038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827846.841040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972859648:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827847.351487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827847.351532:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827847.607535:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827848.375500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827848.375521:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827848.631485:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827849.399435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827849.399450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827849.655485:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827850.423459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827850.423496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827850.679532:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827851.447438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827851.447467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827851.703583:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827851.959499:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827851.959512:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827851.959534:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827851.959544:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4302 00000100:00080000:2.0:1787827851.959553:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827851.959559:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827851.959617:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972859776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827851.959655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827851.959660:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827851.959680:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827851.959687:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4302 00000100:00080000:2.0:1787827851.959693:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827851.959699:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827851.959714:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827851.959750:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972859776:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827851.959751:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972859904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827851.959779:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827851.959784:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827851.959803:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827851.959811:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4302 00000100:00080000:2.0:1787827851.959817:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827851.959823:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827851.959830:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827851.959857:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972860032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827851.959867:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827851.959871:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827851.959873:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827851.959876:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4302 00000100:00080000:2.0:1787827851.959879:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827851.959882:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787827851.959893:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787827851.959895:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827851.959909:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972860160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827851.959917:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827851.959920:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827851.959922:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827851.959926:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4302 00000100:00080000:2.0:1787827851.959928:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827851.959932:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827851.959935:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827851.959946:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972860288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827851.959953:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827851/real 1787827851] req@ffff9c1a49d8f0c0 x1874668972859776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827851.959959:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827851.959983:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827851.959985:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827851.960004:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4302 00000100:00080000:2.0:1787827851.960016:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827851.960030:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827851.960050:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827851.960067:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972860416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827851.960076:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972859776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827851.960089:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827851.960108:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827851.960113:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827851.960118:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827851.960120:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4302 00000100:00080000:3.0:1787827851.960134:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787827851.960135:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827851.960141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972859776:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827851.960149:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827851.960159:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827851.960193:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972859904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827851.960201:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972860544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827851.960222:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827851.960236:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827851/real 1787827851] req@ffff9c1a49d8c780 x1874668972859904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827851.960248:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972859904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827851.960254:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827851.960257:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827851.960259:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972859904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827851.960263:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972860032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827851.960275:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827851.960281:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827851/real 1787827851] req@ffff9c1a49d8de00 x1874668972860032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827851.960289:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972860032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827851.960299:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827851.960301:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827851.960303:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972860032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827851.960307:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972860160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827851.960317:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827851.960323:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827851/real 1787827851] req@ffff9c1a49d8c000 x1874668972860160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827851.960330:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972860160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827851.960336:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827851.960338:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827851.960340:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972860160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827851.960344:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972860288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827851.960355:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827851.960361:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827851/real 1787827851] req@ffff9c1a49d8da40 x1874668972860288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827851.960369:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972860288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827851.960374:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827851.960376:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827851.960379:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972860288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827851.960382:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972860416:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827851.960393:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827851.960399:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827851/real 1787827851] req@ffff9c1a49d8cf00 x1874668972860416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827851.960428:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972860416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827851.960434:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827851.960436:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827851.960438:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972860416:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827851.960456:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972860544:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827851.960469:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827851.960477:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827851/real 1787827851] req@ffff9c1a49d8cb40 x1874668972860544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827851.960485:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972860544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827851.960491:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827851.960494:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827851.960496:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972860544:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827852.471448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827852.471448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827852.727505:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827853.495465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827853.495498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827853.751480:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827854.519500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827854.519522:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827854.775461:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827855.543448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827855.543448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827855.799496:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827856.567433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827856.567433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827856.823549:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827857.079513:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827857.079522:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827857.079527:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827857.079537:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4307 00000100:00080000:0.0:1787827857.079547:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827857.079553:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827857.079592:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972860672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827857.079619:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827857.079624:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827857.079626:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827857.079632:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4307 00000100:00080000:0.0:1787827857.079636:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827857.079643:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827857.079654:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827857.079668:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972860672:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827857.079675:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972860800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827857.079691:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827857.079695:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827857.079697:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827857.079701:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4307 00000100:00080000:0.0:1787827857.079705:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827857.079709:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827857.079714:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827857.079725:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972860928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827857.079734:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827857.079735:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827857.079740:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827857.079741:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827857.079746:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4307 00000100:00080000:0.0:1787827857.079750:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827857.079775:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827857.079779:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827857.079793:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972861056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827857.079798:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827857/real 1787827857] req@ffff9c1a788621c0 x1874668972860672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827857.079806:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827857.079812:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827857.079816:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827857.079822:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4307 00000100:00080000:0.0:1787827857.079829:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827857.079835:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827857.079839:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827857.079845:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972860672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827857.079851:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972861184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827857.079879:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827857.079882:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827857.079884:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827857.079885:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827857.079894:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4307 00000100:00080000:0.0:1787827857.079897:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827857.079899:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827857.079903:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827857.079909:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827857.079911:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972860672:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827857.079920:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972861312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827857.079928:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827857.079930:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827857.079933:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827857.079935:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4307 00000100:00080000:0.0:1787827857.079938:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827857.079941:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827857.079946:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827857.079948:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972860800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827857.079955:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972861440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827857.080001:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827857.080021:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827857/real 1787827857] req@ffff9c1a78860780 x1874668972860800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827857.080033:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972860800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827857.080041:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827857.080044:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827857.080048:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972860800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827857.080053:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972860928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827857.080071:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827857.080080:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827857/real 1787827857] req@ffff9c1a78861e00 x1874668972860928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827857.080090:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972860928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827857.080097:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827857.080099:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827857.080102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972860928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827857.080107:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972861056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827857.080120:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827857.080128:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827857/real 1787827857] req@ffff9c1a78862580 x1874668972861056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827857.080137:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972861056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827857.080145:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827857.080147:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827857.080150:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972861056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827857.080154:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972861184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827857.080167:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827857.080175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827857/real 1787827857] req@ffff9c1a78861680 x1874668972861184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827857.080191:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972861184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827857.080198:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827857.080201:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827857.080203:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972861184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827857.080208:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972861312:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827857.080221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827857.080229:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827857/real 1787827857] req@ffff9c1a788630c0 x1874668972861312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827857.080239:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972861312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827857.080246:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827857.080248:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827857.080251:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972861312:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827857.080266:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972861440:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827857.080280:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827857.080288:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827857/real 1787827857] req@ffff9c1a78863c00 x1874668972861440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827857.080298:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972861440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827857.080327:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827857.080330:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827857.080332:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972861440:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827857.591408:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827857.591437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827857.847606:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827858.615444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827858.615446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827858.871615:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827859.639463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827859.639494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827859.895597:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827860.663453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827860.663458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827860.919485:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827861.687448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827861.687452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827861.943552:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827862.199464:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827862.199474:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827862.199478:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827862.199498:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4312 00000100:00080000:1.0:1787827862.199513:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827862.199521:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827862.199560:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972861568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827862.199586:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827862.199590:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827862.199591:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827862.199595:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4312 00000100:00080000:1.0:1787827862.199598:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827862.199602:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827862.199611:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827862.199622:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972861696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827862.199633:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827862.199636:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827862.199638:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827862.199641:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4312 00000100:00080000:1.0:1787827862.199644:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827862.199652:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827862.199656:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972861568:192.168.203.148@tcp:250:kworker.0 00000008:00100000:1.0:1787827862.199657:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827862.199670:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972861824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827862.199681:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827862.199685:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827862.199688:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827862.199692:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4312 00000100:00080000:1.0:1787827862.199695:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827862.199699:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827862.199702:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827862.199712:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972861952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827862.199741:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827862.199744:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827862.199746:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827862.199750:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4312 00000100:00080000:1.0:1787827862.199755:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827862.199759:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827862.199764:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827862.199770:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827862.199782:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972862080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827862.199792:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827862.199795:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827862.199797:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827862.199802:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4312 00000100:00080000:1.0:1787827862.199805:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827862.199810:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827862.199815:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827862.199838:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972862208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827862.199838:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827862/real 1787827862] req@ffff9c1a78a78f00 x1874668972861568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827862.199903:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827862.199909:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827862.199913:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827862.199916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972861568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827862.199928:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4312 00000100:00080000:1.0:1787827862.199940:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827862.199940:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827862.199952:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787827862.199956:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:1.0:1787827862.199961:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827862.199969:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972861568:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827862.199980:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972862336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827862.200011:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972861696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827862.200042:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827862.200058:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827862/real 1787827862] req@ffff9c1a78a7a580 x1874668972861696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827862.200080:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972861696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827862.200087:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827862.200089:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827862.200092:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972861696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827862.200096:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972861824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827862.200114:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827862.200122:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827862/real 1787827862] req@ffff9c1a78a78000 x1874668972861824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827862.200134:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972861824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827862.200143:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827862.200146:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827862.200150:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972861824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827862.200157:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972861952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827862.200178:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827862.200189:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827862/real 1787827862] req@ffff9c1a78a7b480 x1874668972861952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827862.200198:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972861952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827862.200209:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827862.200211:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827862.200213:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972861952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827862.200218:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972862080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827862.200232:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827862.200238:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827862/real 1787827862] req@ffff9c1a78a792c0 x1874668972862080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827862.200245:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972862080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827862.200251:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827862.200253:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827862.200254:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972862080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827862.200258:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972862208:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827862.200270:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827862.200275:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827862/real 1787827862] req@ffff9c1a78a7b840 x1874668972862208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827862.200283:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972862208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827862.200288:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827862.200290:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827862.200292:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972862208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827862.200297:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972862336:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827862.200307:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827862.200313:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827862/real 1787827862] req@ffff9c1a78a79a40 x1874668972862336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827862.200320:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972862336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827862.200325:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827862.200327:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827862.200329:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972862336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827862.711466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827862.711466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827862.967541:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827863.735487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827863.735487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827863.991555:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827864.759429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827864.759430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827865.015540:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827865.783445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827865.783445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827866.039541:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827866.807429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827866.807431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827867.063521:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827867.319459:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827867.319473:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827867.319480:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827867.319493:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4317 00000100:00080000:1.0:1787827867.319507:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827867.319517:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827867.319579:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972862464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827867.319620:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827867.319626:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827867.319640:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827867.319647:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4317 00000100:00080000:1.0:1787827867.319653:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827867.319661:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827867.319676:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827867.319723:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972862464:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827867.319733:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972862592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827867.319757:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827867.319764:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827867.319768:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827867.319775:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4317 00000100:00080000:1.0:1787827867.319781:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827867.319792:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827867.319801:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827867.319816:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827867.319825:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972862720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827867.319856:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827867.319862:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827867.319865:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827867.319874:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4317 00000100:00080000:1.0:1787827867.319880:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827867.319883:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827867/real 1787827867] req@ffff9c1a78a79a40 x1874668972862464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827867.319892:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827867.319909:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827867.319923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972862464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827867.319934:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972862848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827867.319942:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827867.319959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827867.319969:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827867.319971:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972862464:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827867.319984:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827867.319990:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827867.319998:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4317 00000100:00080000:1.0:1787827867.320004:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827867.320011:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827867.320015:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972862592:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787827867.320022:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827867.320048:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972862976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827867.320059:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827867.320079:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827867.320084:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827867.320087:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827867.320097:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827867/real 1787827867] req@ffff9c1a78a7b840 x1874668972862592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827867.320098:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4317 00000100:00080000:1.0:1787827867.320105:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827867.320117:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827867.320133:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827867.320142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972862592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827867.320156:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972863104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827867.320159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827867.320173:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827867.320186:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972862592:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827867.320200:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827867.320206:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972862720:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827867.320226:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827867.320231:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827867.320238:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827867.320247:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4317 00000100:00080000:1.0:1787827867.320257:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827867.320259:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827867/real 1787827867] req@ffff9c1a78a792c0 x1874668972862720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827867.320269:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827867.320283:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827867.320301:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972862720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827867.320310:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972863232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827867.320335:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827867.320424:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827867.320431:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972862720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827867.320439:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972862848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827867.320490:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827867.320513:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827867/real 1787827867] req@ffff9c1a78a7b480 x1874668972862848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827867.320535:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972862848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827867.320551:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827867.320557:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827867.320562:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972862848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827867.320586:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972862976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827867.320619:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827867.320632:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827867/real 1787827867] req@ffff9c1a78a78000 x1874668972862976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827867.320644:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972862976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827867.320654:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827867.320657:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827867.320660:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972862976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827867.320666:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972863104:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827867.320709:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827867.320733:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827867/real 1787827867] req@ffff9c1a78a7a580 x1874668972863104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827867.320754:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972863104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827867.320763:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827867.320766:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827867.320770:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972863104:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827867.320776:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972863232:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827867.320804:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827867.320813:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827867/real 1787827867] req@ffff9c1a78a78f00 x1874668972863232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827867.320825:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972863232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827867.320835:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827867.320850:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827867.320856:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972863232:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827867.831542:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827867.831553:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827868.087532:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827868.855407:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827868.855407:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827869.111450:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827869.879435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827869.879444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827870.135484:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827870.903467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827870.903501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827871.159600:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827871.927458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827871.927458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827872.183709:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827872.439463:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827872.439473:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827872.439477:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827872.439487:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4322 00000100:00080000:1.0:1787827872.439498:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827872.439506:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827872.439561:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972863360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827872.439590:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827872.439593:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827872.439595:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827872.439598:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4322 00000100:00080000:1.0:1787827872.439601:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827872.439605:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827872.439630:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827872.439641:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972863488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827872.439660:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972863360:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827872.439674:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827872.439677:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827872.439679:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827872.439683:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4322 00000100:00080000:1.0:1787827872.439687:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827872.439690:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827872.439695:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827872.439704:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972863616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827872.439711:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827872.439714:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827872.439715:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827872.439720:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4322 00000800:00000100:3.0:1787827872.439720:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827872.439724:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827872.439729:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827872.439733:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827872.439741:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972863744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827872.439747:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827872.439750:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827872.439751:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827872.439754:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4322 00000100:00080000:1.0:1787827872.439757:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827872.439762:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827872.439764:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827872/real 1787827872] req@ffff9c1a78a7a940 x1874668972863360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827872.439765:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827872.439775:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972863872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827872.439788:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827872.439790:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827872.439791:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827872.439796:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4322 00000100:00100000:3.0:1787827872.439798:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972863360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827872.439799:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827872.439805:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827872.439810:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827872.439819:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827872.439821:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972864000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827872.439827:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827872.439833:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827872.439835:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972863360:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827872.439838:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827872.439843:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827872.439847:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4322 00000100:00080000:1.0:1787827872.439849:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827872.439852:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827872.439855:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827872.439862:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972864128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827872.439874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972863488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827872.439897:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827872.439908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827872/real 1787827872] req@ffff9c1a78a783c0 x1874668972863488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827872.439916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972863488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827872.439922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827872.439925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827872.439927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972863488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827872.439930:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972863616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827872.439941:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827872.439947:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827872/real 1787827872] req@ffff9c1a78a7a1c0 x1874668972863616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827872.439954:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972863616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827872.439964:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827872.439966:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827872.439968:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972863616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827872.439971:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972863744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827872.439980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827872.439987:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827872/real 1787827872] req@ffff9c1a78a78780 x1874668972863744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827872.439998:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972863744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827872.440006:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827872.440010:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827872.440013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972863744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827872.440019:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972863872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827872.440039:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827872.440049:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827872/real 1787827872] req@ffff9c1a78a7bc00 x1874668972863872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827872.440057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972863872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827872.440063:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827872.440065:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827872.440067:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972863872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827872.440072:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972864000:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827872.440084:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827872.440090:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827872/real 1787827872] req@ffff9c1a78a7b0c0 x1874668972864000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827872.440102:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972864000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827872.440110:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827872.440113:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827872.440117:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972864000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827872.440136:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972864128:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827872.440162:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827872.440171:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827872/real 1787827872] req@ffff9c1a78a79e00 x1874668972864128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827872.440179:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972864128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827872.440185:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827872.440187:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827872.440189:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972864128:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827872.951412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827872.951435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827873.207544:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827873.975435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827873.975435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827874.231699:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827874.999439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827874.999474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827875.255560:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827876.023485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827876.023488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827876.279663:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827877.047427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827877.047429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827877.303450:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827877.559461:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827877.559469:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827877.559472:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827877.559481:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4327 00000100:00080000:0.0:1787827877.559490:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827877.559496:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827877.559528:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972864256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827877.559551:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827877.559554:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827877.559555:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827877.559558:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4327 00000100:00080000:0.0:1787827877.559560:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827877.559563:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827877.559571:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827877.559581:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972864384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827877.559589:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827877.559591:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827877.559594:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827877.559596:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972864256:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827877.559599:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4327 00000100:00080000:0.0:1787827877.559605:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827877.559608:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827877.559630:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827877.559641:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972864512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827877.559650:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827877.559653:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827877.559655:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827877.559657:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4327 00000100:00080000:0.0:1787827877.559661:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827877.559664:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827877.559668:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827877.559675:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827877.559677:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972864640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827877.559686:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827877.559689:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827877.559690:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827877.559693:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4327 00000100:00080000:0.0:1787827877.559695:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827877.559698:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827877.559701:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827877.559709:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972864768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827877.559715:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827877.559717:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827877.559718:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827877.559722:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4327 00000100:00100000:3.0:1787827877.559723:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827877/real 1787827877] req@ffff9c1a78860000 x1874668972864256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827877.559725:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827877.559732:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827877.559739:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827877.559750:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972864896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827877.559753:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972864256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827877.559771:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827877.559774:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827877.559776:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827877.559777:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827877.559784:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827877.559785:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4327 00000100:00080000:0.0:1787827877.559790:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827877.559791:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972864256:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827877.559795:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827877.559801:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827877.559809:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972865024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827877.559820:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972864384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827877.559842:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827877.559853:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827877/real 1787827877] req@ffff9c1a78863c00 x1874668972864384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827877.559861:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972864384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827877.559880:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827877.559884:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827877.559888:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972864384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827877.559894:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972864512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827877.559925:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827877.559936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827877/real 1787827877] req@ffff9c1a788630c0 x1874668972864512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827877.559945:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972864512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827877.559951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827877.559953:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827877.559955:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972864512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827877.559959:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972864640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827877.559973:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827877.559980:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827877/real 1787827877] req@ffff9c1a78861680 x1874668972864640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827877.559987:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972864640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827877.560003:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827877.560004:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827877.560006:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972864640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827877.560010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972864768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827877.560023:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827877.560034:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827877/real 1787827877] req@ffff9c1a78862580 x1874668972864768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827877.560053:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972864768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827877.560063:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827877.560066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827877.560070:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972864768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827877.560076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972864896:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827877.560105:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827877.560114:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827877/real 1787827877] req@ffff9c1a78861e00 x1874668972864896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827877.560122:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972864896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827877.560127:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827877.560129:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827877.560131:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972864896:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827877.560144:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972865024:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827877.560156:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827877.560162:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827877/real 1787827877] req@ffff9c1a78860780 x1874668972865024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827877.560169:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972865024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827877.560178:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827877.560180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827877.560182:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972865024:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827878.071434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827878.071438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827878.327602:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827879.095462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827879.095483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827879.351651:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827880.119460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827880.119460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827880.375555:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827881.143394:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827881.143466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827881.399517:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827882.167424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827882.167424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827882.423491:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827882.679476:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827882.679487:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827882.679492:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827882.679502:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4333 00000100:00080000:0.0:1787827882.679514:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827882.679522:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827882.679564:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972865152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827882.679601:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827882.679605:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827882.679606:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827882.679610:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4333 00000100:00080000:0.0:1787827882.679613:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827882.679617:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827882.679627:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827882.679637:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972865280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827882.679646:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827882.679648:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827882.679650:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827882.679654:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4333 00000100:00080000:0.0:1787827882.679660:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827882.679663:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972865152:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827882.679666:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827882.679678:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827882.679687:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972865408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827882.679697:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827882.679700:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827882.679701:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827882.679703:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4333 00000100:00080000:0.0:1787827882.679707:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827882.679710:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827882.679713:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827882.679722:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972865536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827882.679733:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827882.679735:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827882.679737:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827882.679739:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4333 00000100:00080000:0.0:1787827882.679741:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827882.679747:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787827882.679751:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787827882.679753:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827882.679765:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972865664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827882.679773:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827882.679775:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827882.679777:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827882.679780:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4333 00000100:00080000:0.0:1787827882.679782:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827882.679784:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827882.679789:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827882.679799:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972865792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827882.679806:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827882.679812:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787827882.679814:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827882/real 1787827882] req@ffff9c1a788621c0 x1874668972865152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827882.679815:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827882.679825:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4333 00000100:00080000:0.0:1787827882.679833:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827882.679840:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827882.679848:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827882.679862:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972865920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827882.679874:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972865152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827882.679889:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827882.679894:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827882.679900:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972865152:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827882.679932:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972865280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827882.679966:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827882.679990:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827882/real 1787827882] req@ffff9c1a78862940 x1874668972865280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827882.680012:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972865280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827882.680024:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827882.680029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827882.680043:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972865280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827882.680052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972865408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827882.680083:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827882.680111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827882/real 1787827882] req@ffff9c1a78861a40 x1874668972865408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827882.680128:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972865408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827882.680139:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827882.680145:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827882.680150:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972865408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827882.680160:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972865536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827882.680263:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827882.680278:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827882/real 1787827882] req@ffff9c1a78860f00 x1874668972865536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827882.680295:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972865536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827882.680311:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827882.680323:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827882.680464:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972865536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827882.680473:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972865664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827882.680552:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827882.680567:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827882/real 1787827882] req@ffff9c1a78862d00 x1874668972865664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827882.680589:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972865664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827882.680596:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827882.680598:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827882.680601:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972865664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827882.680606:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972865792:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827882.680624:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827882.680631:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827882/real 1787827882] req@ffff9c1a78860b40 x1874668972865792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827882.680639:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972865792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827882.680644:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827882.680646:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827882.680649:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972865792:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827882.680653:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972865920:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827882.680663:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827882.680668:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827882/real 1787827882] req@ffff9c1a78863840 x1874668972865920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827882.680675:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972865920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827882.680681:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827882.680683:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827882.680685:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972865920:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827883.191454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827883.191504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827883.447524:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827884.215425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827884.215449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827884.471480:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827885.239437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827885.239438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827885.495572:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827886.263435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827886.263437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827886.519498:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827887.287445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827887.287455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827887.543552:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827887.799435:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827887.799443:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827887.799446:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827887.799455:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4338 00000100:00080000:0.0:1787827887.799470:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827887.799476:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827887.799509:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972866048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827887.799530:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827887.799533:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827887.799534:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827887.799537:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4338 00000100:00080000:0.0:1787827887.799553:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827887.799557:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827887.799566:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827887.799577:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972866048:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827887.799581:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972866176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827887.799590:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827887.799593:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827887.799594:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827887.799597:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4338 00000100:00080000:0.0:1787827887.799599:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827887.799601:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827887.799605:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827887.799613:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972866304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827887.799620:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827887.799623:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827887.799624:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827887.799626:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4338 00000100:00080000:0.0:1787827887.799628:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827887.799630:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787827887.799634:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787827887.799635:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827887.799645:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972866432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827887.799652:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827887.799654:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827887.799656:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827887.799659:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4338 00000100:00080000:0.0:1787827887.799662:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827887.799664:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827887.799667:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827887.799669:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827887/real 1787827887] req@ffff9c1a788612c0 x1874668972866048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827887.799677:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972866560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827887.799685:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827887.799687:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827887.799690:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827887.799691:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972866048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827887.799695:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4338 00000100:00080000:0.0:1787827887.799700:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827887.799705:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787827887.799708:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827887.799709:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827887.799716:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827887.799720:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972866688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827887.799722:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972866048:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827887.799735:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827887.799737:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827887.799738:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827887.799742:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4338 00000100:00080000:0.0:1787827887.799744:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827887.799747:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827887.799750:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827887.799751:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972866176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827887.799760:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972866816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827887.799777:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827887.799788:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827887/real 1787827887] req@ffff9c1a78863840 x1874668972866176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.799796:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972866176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.799802:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.799804:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827887.799807:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972866176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827887.799811:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972866304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827887.799821:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827887.799827:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827887/real 1787827887] req@ffff9c1a78860b40 x1874668972866304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.799834:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972866304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.799839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.799840:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827887.799842:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972866304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827887.799846:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972866432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827887.799855:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827887.799861:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827887/real 1787827887] req@ffff9c1a78862d00 x1874668972866432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.799873:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972866432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.799878:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.799879:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827887.799881:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972866432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827887.799884:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972866560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827887.799894:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827887.799899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827887/real 1787827887] req@ffff9c1a78860f00 x1874668972866560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.799906:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972866560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.799911:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.799913:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827887.799914:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972866560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827887.799918:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972866688:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827887.799928:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827887.799933:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827887/real 1787827887] req@ffff9c1a78861a40 x1874668972866688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.799940:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972866688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.799949:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.799951:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827887.799953:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972866688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827887.799964:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972866816:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827887.799973:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827887.799979:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827887/real 1787827887] req@ffff9c1a78862940 x1874668972866816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827887.799986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972866816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827887.799991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827887.799993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827887.799995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972866816:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827888.311429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827888.311432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827888.567599:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827889.335442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827889.335442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827889.591554:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827890.359441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827890.359441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827890.615483:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827891.383451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827891.383460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827891.639463:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827892.407412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827892.407412:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827892.663522:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827892.919463:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827892.919475:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827892.919479:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827892.919488:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4343 00000100:00080000:1.0:1787827892.919499:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827892.919519:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827892.919559:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972866944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827892.919588:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827892.919593:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827892.919594:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827892.919599:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4343 00000100:00080000:1.0:1787827892.919604:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827892.919609:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827892.919618:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827892.919625:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972866944:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827892.919633:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972867072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827892.919647:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827892.919651:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827892.919653:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827892.919657:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4343 00000100:00080000:1.0:1787827892.919662:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827892.919667:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827892.919672:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827892.919681:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827892.919687:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972867200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827892.919701:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827892.919704:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827892.919706:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827892.919711:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4343 00000100:00080000:1.0:1787827892.919715:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827892.919721:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827892/real 1787827892] req@ffff9c1a78a79e00 x1874668972866944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827892.919723:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827892.919733:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827892.919753:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972866944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827892.919754:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972867328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827892.919776:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827892.919779:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827892.919783:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827892.919787:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827892.919790:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972866944:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827892.919793:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827892.919800:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4343 00000100:00080000:1.0:1787827892.919805:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827892.919810:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827892.919813:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972867072:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787827892.919820:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827892.919838:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972867456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827892.919841:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827892.919854:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827892/real 1787827892] req@ffff9c1a78a7b0c0 x1874668972867072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827892.919856:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827892.919866:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827892.919871:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827892.919875:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972867072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827892.919880:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4343 00000100:00080000:3.0:1787827892.919886:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827892.919889:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827892.919891:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827892.919899:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827892.919905:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972867072:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787827892.919911:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827892.919912:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972867200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827892.919952:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972867584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827892.919957:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827892.919971:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827892/real 1787827892] req@ffff9c1a78a7bc00 x1874668972867200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827892.919972:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827892.920001:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827892.920006:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827892.920011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972867200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827892.920016:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4343 00000100:00080000:3.0:1787827892.920022:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827892.920025:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827892.920028:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827892.920033:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972867200:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827892.920036:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827892.920042:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972867328:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787827892.920047:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827892.920059:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827892.920066:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972867712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827892.920070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827892/real 1787827892] req@ffff9c1a78a78780 x1874668972867328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827892.920098:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972867328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827892.920106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827892.920109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827892.920111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972867328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827892.920124:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972867456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827892.920136:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827892.920142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827892/real 1787827892] req@ffff9c1a78a7a1c0 x1874668972867456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827892.920149:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972867456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827892.920154:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827892.920156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827892.920158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972867456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827892.920162:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972867584:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827892.920172:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827892.920177:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827892/real 1787827892] req@ffff9c1a78a783c0 x1874668972867584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827892.920184:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972867584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827892.920194:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827892.920196:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827892.920198:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972867584:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827892.920201:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972867712:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827892.920210:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827892.920216:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827892/real 1787827892] req@ffff9c1a78a7a940 x1874668972867712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827892.920222:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972867712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827892.920227:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827892.920229:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827892.920231:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972867712:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827893.431421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827893.431421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827893.687533:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827894.455429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827894.455473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827894.711523:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827895.479460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827895.479466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827895.735527:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827896.503425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827896.503472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827896.759520:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827897.527434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827897.527496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827897.783541:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827898.039535:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827898.039545:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827898.039549:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827898.039557:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4348 00000100:00080000:0.0:1787827898.039567:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827898.039573:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827898.039609:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972867840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827898.039634:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827898.039637:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827898.039639:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827898.039641:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4348 00000100:00080000:0.0:1787827898.039644:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827898.039647:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827898.039655:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827898.039664:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972867968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827898.039672:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827898.039674:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827898.039676:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827898.039681:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972867840:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827898.039682:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4348 00000100:00080000:0.0:1787827898.039692:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827898.039697:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827898.039703:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827898.039716:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972868096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827898.039727:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827898.039731:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827898.039733:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827898.039737:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4348 00000100:00080000:0.0:1787827898.039741:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827898.039743:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827898.039751:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827898.039757:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827898.039769:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972868224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827898.039780:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827898.039783:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827898.039784:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827898.039786:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4348 00000100:00080000:0.0:1787827898.039790:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827898.039792:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827898/real 1787827898] req@ffff9c1a788621c0 x1874668972867840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827898.039796:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827898.039802:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827898.039815:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972868352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827898.039819:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972867840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827898.039835:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827898.039837:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827898.039841:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827898.039842:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827898.039845:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827898.039848:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972867840:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827898.039850:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4348 00000100:00080000:0.0:1787827898.039855:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827898.039859:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827898.039863:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827898.039870:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972868480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827898.039878:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827898.039883:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787827898.039883:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972867968:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827898.039887:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827898.039894:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4348 00000100:00080000:0.0:1787827898.039897:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827898.039900:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827898.039903:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827898.039912:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972868608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827898.039919:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827898.039936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827898/real 1787827898] req@ffff9c1a78860780 x1874668972867968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827898.039948:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972867968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827898.039954:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827898.039956:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827898.039958:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972867968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827898.039963:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972868096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827898.039974:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827898.039980:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827898/real 1787827898] req@ffff9c1a78861e00 x1874668972868096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827898.039987:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972868096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827898.039992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827898.039995:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827898.039996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972868096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827898.040000:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972868224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827898.040010:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827898.040016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827898/real 1787827898] req@ffff9c1a78862580 x1874668972868224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827898.040023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972868224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827898.040028:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827898.040030:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827898.040032:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972868224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827898.040035:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972868352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827898.040045:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827898.040050:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827898/real 1787827898] req@ffff9c1a78861680 x1874668972868352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827898.040057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972868352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827898.040062:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827898.040064:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827898.040066:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972868352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827898.040076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972868480:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827898.040087:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827898.040092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827898/real 1787827898] req@ffff9c1a788630c0 x1874668972868480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827898.040100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972868480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827898.040105:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827898.040107:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827898.040109:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972868480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827898.040113:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972868608:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827898.040121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827898.040126:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827898/real 1787827898] req@ffff9c1a78863c00 x1874668972868608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827898.040133:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972868608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827898.040138:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827898.040140:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827898.040142:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972868608:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827898.551425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827898.551426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827898.807435:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827899.575479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827899.575490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827899.831521:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827900.599479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827900.599480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827900.855519:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827901.623493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827901.623497:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827901.879523:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827902.647487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827902.647496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827902.903518:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827903.159459:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827903.159469:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827903.159474:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827903.159483:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4353 00000100:00080000:3.0:1787827903.159494:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827903.159501:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827903.159540:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972868736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827903.159574:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827903.159578:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827903.159580:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827903.159584:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4353 00000100:00080000:3.0:1787827903.159588:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827903.159592:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827903.159601:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827903.159617:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972868864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827903.159623:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972868736:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827903.159637:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827903.159640:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827903.159642:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827903.159647:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4353 00000100:00080000:3.0:1787827903.159651:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827903.159655:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827903.159669:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827903.159684:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972868992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827903.159695:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827903.159699:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827903.159701:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827903.159705:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4353 00000100:00080000:3.0:1787827903.159711:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827903.159716:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827903.159722:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827903.159734:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972868864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827903.159737:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972869120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827903.159753:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827903.159757:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827903.159760:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827903.159769:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4353 00000100:00100000:2.0:1787827903.159775:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972868992:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827903.159775:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827903.159783:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827903.159790:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827903.159805:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972869248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827903.159813:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972869120:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827903.159820:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827903.159825:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827903.159827:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827903.159832:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4353 00000100:00080000:3.0:1787827903.159845:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827903.159850:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972869248:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827903.159854:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827903.159861:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827903.159876:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972869376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827903.159888:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827903.159893:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972869376:192.168.203.148@tcp:8:kworker.0 00000100:00080000:3.0:1787827903.159893:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827903.159897:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827903.159905:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4353 00000100:00080000:3.0:1787827903.159917:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827903.159922:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827903.159930:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827903.159949:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972869504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827903.159992:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972869504:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827903.671427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827903.671449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827903.927525:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827904.695467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827904.695467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827904.951537:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827905.719441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827905.719441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827905.975476:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827906.231599:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787827906.231637:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827906.231651:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827906.231687:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827906.231693:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827906.231700:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827906.231707:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827906.231713:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827906.231722:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787827906.231746:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827903/real 1787827906] req@ffff9c1a7895c000 x1874668972869504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.231773:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972869504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.231786:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.231793:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827906.231798:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972869504:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827906.231818:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827903/real 1787827906] req@ffff9c1a7895f480 x1874668972869376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.231830:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972869376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.231839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.231843:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827906.231847:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972869376:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827906.231853:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827903/real 1787827906] req@ffff9c1a7895cf00 x1874668972869248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.231864:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972869248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.231871:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.231875:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827906.231878:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972869248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827906.231884:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827903/real 1787827906] req@ffff9c1a7895c3c0 x1874668972869120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.231895:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972869120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.231903:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.231906:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827906.231909:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972869120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827906.231915:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827903/real 1787827906] req@ffff9c1a7895e580 x1874668972868864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.231927:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972868864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.231941:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.231945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827906.231948:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972868864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827906.231955:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827903/real 1787827906] req@ffff9c1a7895fc00 x1874668972868992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.231969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972868992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.231977:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.231981:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827906.231984:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972868992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827906.231991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827903/real 1787827906] req@ffff9c1a7895e940 x1874668972868736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.232002:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972868736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.232009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.232014:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827906.232017:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972868736:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827906.743476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827906.743477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827906.999505:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827907.767439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827907.767444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827908.023515:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827908.279497:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827908.279506:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827908.279510:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827908.279518:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4358 00000100:00080000:0.0:1787827908.279527:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827908.279534:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827908.279563:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972869632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827908.279582:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827908.279585:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827908.279586:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827908.279589:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4358 00000100:00080000:0.0:1787827908.279592:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827908.279595:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827908.279603:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827908.279613:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972869760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827908.279621:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827908.279625:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827908.279628:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827908.279629:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972869632:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827908.279632:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4358 00000100:00080000:0.0:1787827908.279636:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827908.279642:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827908.279645:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827908.279652:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972869888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827908.279665:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827908.279668:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827908.279670:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827908.279674:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4358 00000100:00080000:0.0:1787827908.279678:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827908.279682:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827908.279696:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827908.279702:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827908.279713:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972870016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827908.279724:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827908.279727:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827908.279728:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827908.279731:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4358 00000100:00080000:0.0:1787827908.279735:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827908.279739:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827908.279743:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827908.279745:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827908/real 1787827908] req@ffff9c1a78860000 x1874668972869632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827908.279758:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972870144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827908.279771:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972869632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827908.279777:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827908.279784:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827908.279786:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827908.279787:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827908.279793:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4358 00000100:00080000:2.0:1787827908.279798:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827908.279801:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827908.279806:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972869632:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827908.279811:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827908.279820:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827908.279832:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972870272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827908.279840:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972869760:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827908.279851:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827908.279856:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827908.279858:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827908.279864:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4358 00000100:00080000:0.0:1787827908.279869:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827908.279873:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787827908.279878:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787827908.279883:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827908.279900:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827908/real 1787827908] req@ffff9c1a78863c00 x1874668972869760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827908.279901:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972870400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827908.279938:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972869760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827908.279949:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827908.279954:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827908.279959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972869760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827908.279970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972869888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827908.279998:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827908.280010:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827908/real 1787827908] req@ffff9c1a788630c0 x1874668972869888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827908.280023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972869888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827908.280031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827908.280034:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827908.280037:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972869888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827908.280044:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972870016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827908.280065:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827908.280077:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827908/real 1787827908] req@ffff9c1a78861680 x1874668972870016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827908.280089:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972870016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827908.280097:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827908.280101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827908.280104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972870016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827908.280110:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972870144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827908.280128:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827908.280140:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827908/real 1787827908] req@ffff9c1a78862580 x1874668972870144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827908.280152:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972870144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827908.280160:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827908.280163:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827908.280167:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972870144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827908.280188:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972870272:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827908.280209:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827908.280221:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827908/real 1787827908] req@ffff9c1a78861e00 x1874668972870272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827908.280234:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972870272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827908.280242:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827908.280246:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827908.280249:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972870272:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827908.280255:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972870400:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827908.280282:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827908.280294:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827908/real 1787827908] req@ffff9c1a78860780 x1874668972870400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827908.280307:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972870400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827908.280322:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827908.280326:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827908.280330:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972870400:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827908.791460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827908.791461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827909.047498:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827909.815435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827909.815486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827910.071462:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827910.839429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827910.839473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827911.095502:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827911.863431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827911.863451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827912.119513:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827912.887449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827912.887481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827913.143515:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827913.399450:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827913.399465:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827913.399469:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827913.399477:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4363 00000100:00080000:3.0:1787827913.399487:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827913.399493:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827913.399528:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972870528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827913.399549:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827913.399551:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827913.399552:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827913.399555:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4363 00000100:00080000:3.0:1787827913.399557:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827913.399560:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827913.399567:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827913.399576:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972870656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827913.399583:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827913.399585:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827913.399587:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827913.399589:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4363 00000100:00080000:3.0:1787827913.399591:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827913.399593:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827913.399596:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827913.399602:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972870784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827913.399609:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972870528:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827913.399612:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827913.399618:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827913.399621:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827913.399625:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4363 00000100:00080000:3.0:1787827913.399627:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827913.399629:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827913.399632:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827913.399640:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972870912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827913.399646:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827913.399648:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827913.399649:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827913.399652:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4363 00000100:00080000:3.0:1787827913.399654:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827913.399657:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827913.399659:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827913.399666:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972871040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827913.399671:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827913.399673:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827913.399674:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827913.399677:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4363 00000100:00080000:3.0:1787827913.399679:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827913.399681:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827913.399685:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827913.399691:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827913.399693:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972871168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827913.399700:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827913.399703:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827913.399704:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827913.399707:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4363 00000100:00080000:3.0:1787827913.399709:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827913.399712:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827913.399714:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827913.399720:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972871296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827913.399748:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827913/real 1787827913] req@ffff9c1a7895f840 x1874668972870528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827913.399770:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972870528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827913.399783:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827913.399789:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827913.399794:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972870528:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827913.399803:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972870656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827913.399830:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827913.399846:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827913/real 1787827913] req@ffff9c1a7895f0c0 x1874668972870656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827913.399879:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972870656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827913.399889:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827913.399893:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827913.399897:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972870656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827913.399925:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972870784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827913.399949:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827913.399961:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827913/real 1787827913] req@ffff9c1a7895da40 x1874668972870784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827913.399975:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972870784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827913.399981:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827913.399984:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827913.399986:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972870784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827913.399990:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972870912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827913.400005:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827913.400013:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827913/real 1787827913] req@ffff9c1a7895e1c0 x1874668972870912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827913.400020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972870912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827913.400030:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827913.400033:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827913.400034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972870912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827913.400038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972871040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827913.400047:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827913.400053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827913/real 1787827913] req@ffff9c1a7895de00 x1874668972871040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827913.400061:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972871040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827913.400068:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827913.400070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827913.400072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972871040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827913.400076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972871168:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827913.400089:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827913.400095:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827913/real 1787827913] req@ffff9c1a7895c780 x1874668972871168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827913.400102:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972871168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827913.400107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827913.400108:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827913.400111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972871168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827913.400114:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972871296:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827913.400124:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827913.400129:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827913/real 1787827913] req@ffff9c1a7895cb40 x1874668972871296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827913.400136:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972871296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827913.400142:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827913.400143:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827913.400145:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972871296:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827913.911422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827913.911436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827914.167496:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827914.935426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827914.935432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827915.191523:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827915.959423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827915.959454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827916.215514:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827916.983435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827916.983455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827917.239470:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827918.007413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827918.007416:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827918.263457:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827918.519414:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827918.519422:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827918.519443:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827918.519451:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4368 00000100:00080000:3.0:1787827918.519460:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827918.519466:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827918.519499:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972871424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827918.519523:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827918.519526:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827918.519528:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827918.519531:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4368 00000100:00080000:3.0:1787827918.519533:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827918.519536:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827918.519543:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827918.519553:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972871552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827918.519561:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827918.519563:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827918.519565:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827918.519568:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4368 00000100:00080000:3.0:1787827918.519570:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827918.519573:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827918.519577:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827918.519580:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972871424:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827918.519587:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972871680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827918.519604:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827918.519606:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827918.519608:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827918.519610:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4368 00000100:00080000:3.0:1787827918.519613:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827918.519615:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827918.519618:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827918.519627:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972871808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827918.519654:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827918.519656:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827918.519658:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827918.519660:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4368 00000100:00080000:3.0:1787827918.519663:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827918.519666:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827918.519669:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827918.519674:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827918.519682:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972871936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827918.519689:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827918.519691:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827918.519693:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827918.519695:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4368 00000100:00080000:3.0:1787827918.519698:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827918.519700:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827918.519704:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827918.519715:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972872064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827918.519717:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827918/real 1787827918] req@ffff9c1a7895d2c0 x1874668972871424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827918.519748:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827918.519755:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827918.519759:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827918.519762:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4368 00000100:00080000:3.0:1787827918.519765:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827918.519768:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827918.519771:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827918.519775:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972871424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827918.519782:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972872192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827918.519817:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827918.519824:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827918.519831:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972871424:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827918.519840:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972871552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827918.519865:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827918.519881:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827918/real 1787827918] req@ffff9c1a7895cb40 x1874668972871552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827918.519895:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972871552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827918.519905:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827918.519910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827918.519914:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972871552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827918.519940:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972871680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827918.519960:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827918.519968:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827918/real 1787827918] req@ffff9c1a7895c780 x1874668972871680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827918.519978:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972871680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827918.519984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827918.519987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827918.519991:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972871680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827918.519997:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972871808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827918.520013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827918.520021:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827918/real 1787827918] req@ffff9c1a7895de00 x1874668972871808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827918.520029:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972871808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827918.520034:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827918.520036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827918.520039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972871808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827918.520043:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972871936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827918.520053:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827918.520063:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827918/real 1787827918] req@ffff9c1a7895e1c0 x1874668972871936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827918.520076:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972871936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827918.520084:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827918.520088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827918.520093:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972871936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827918.520100:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972872064:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827918.520120:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827918.520132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827918/real 1787827918] req@ffff9c1a7895da40 x1874668972872064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827918.520144:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972872064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827918.520152:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827918.520155:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827918.520158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972872064:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827918.520165:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972872192:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827918.520184:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827918.520196:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827918/real 1787827918] req@ffff9c1a7895f0c0 x1874668972872192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827918.520209:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972872192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827918.520217:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827918.520220:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827918.520223:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972872192:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827919.031426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827919.031434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827919.287468:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827920.055416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827920.055421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827920.311426:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827921.079441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827921.079441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827921.335444:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827922.103452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827922.103452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827922.359466:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827923.127440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827923.127443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827923.383608:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827923.639545:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827923.639556:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827923.639560:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827923.639571:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4373 00000100:00080000:0.0:1787827923.639584:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827923.639591:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827923.639648:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972872320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827923.639679:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827923.639683:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827923.639685:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827923.639689:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4373 00000100:00080000:0.0:1787827923.639692:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827923.639706:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827923.639716:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827923.639729:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972872448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827923.639740:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827923.639743:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827923.639746:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827923.639749:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4373 00000100:00080000:0.0:1787827923.639752:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827923.639759:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827923.639773:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827923.639773:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972872320:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827923.639806:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972872576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827923.639818:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827923.639821:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827923.639823:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827923.639827:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4373 00000100:00080000:0.0:1787827923.639831:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827923.639835:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827923.639840:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827923.639852:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972872704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827923.639861:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827923.639864:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827923.639866:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827923.639884:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827923.639887:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4373 00000100:00080000:0.0:1787827923.639895:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827923.639899:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827923.639905:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827923.639920:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972872832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827923.639930:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827923.639934:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827923.639947:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827923/real 1787827923] req@ffff9c1a788621c0 x1874668972872320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827923.639948:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827923.639961:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4373 00000100:00080000:0.0:1787827923.639969:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827923.639979:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827923.639989:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827923.640014:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972872320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827923.640020:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972872960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827923.640037:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827923.640040:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827923.640045:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827923.640053:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4373 00000100:00080000:2.0:1787827923.640055:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827923.640058:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827923.640067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827923.640073:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827923.640079:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972872320:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787827923.640105:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827923.640127:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972873088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827923.640133:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972872448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827923.640191:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827923.640211:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827923/real 1787827923] req@ffff9c1a78862940 x1874668972872448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827923.640226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972872448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827923.640235:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827923.640249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827923.640262:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972872448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827923.640269:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972872576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827923.640314:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827923.640354:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827923/real 1787827923] req@ffff9c1a78861a40 x1874668972872576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827923.640413:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972872576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827923.640437:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827923.640441:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827923.640444:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972872576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827923.640451:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972872704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827923.640476:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827923.640499:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827923/real 1787827923] req@ffff9c1a78860f00 x1874668972872704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827923.640511:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972872704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827923.640519:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827923.640522:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827923.640525:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972872704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827923.640531:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972872832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827923.640548:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827923.640558:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827923/real 1787827923] req@ffff9c1a78862d00 x1874668972872832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827923.640569:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972872832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827923.640576:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827923.640579:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827923.640582:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972872832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827923.640588:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972872960:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827923.640603:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827923.640611:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827923/real 1787827923] req@ffff9c1a78860b40 x1874668972872960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827923.640622:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972872960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827923.640636:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827923.640639:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827923.640641:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972872960:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827923.640661:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972873088:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827923.640676:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827923.640686:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827923/real 1787827923] req@ffff9c1a78863840 x1874668972873088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827923.640696:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972873088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827923.640715:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827923.640718:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827923.640721:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972873088:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827924.151481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827924.151481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827924.407503:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827925.175481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827925.175486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827925.431603:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827926.199477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827926.199477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827926.455601:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827927.223446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827927.223463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827927.479524:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827928.247514:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827928.247515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827928.503530:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827928.759461:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827928.759470:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827928.759474:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827928.759484:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4379 00000100:00080000:3.0:1787827928.759494:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827928.759500:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827928.759536:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972873216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827928.759561:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827928.759564:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827928.759565:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827928.759569:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4379 00000100:00080000:3.0:1787827928.759571:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827928.759590:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827928.759598:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827928.759608:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972873344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827928.759617:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827928.759619:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827928.759620:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827928.759625:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972873216:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827928.759626:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4379 00000100:00080000:3.0:1787827928.759631:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827928.759636:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827928.759654:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827928.759661:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972873472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827928.759668:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827928.759670:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827928.759672:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827928.759676:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4379 00000100:00080000:3.0:1787827928.759679:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827928.759681:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827928.759685:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827928.759694:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972873600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827928.759700:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827928.759702:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827928.759720:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827928.759723:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4379 00000100:00080000:3.0:1787827928.759726:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827928.759728:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787827928.759732:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787827928.759733:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827928.759744:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972873728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827928.759751:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827928.759754:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827928.759755:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827928.759758:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4379 00000100:00080000:3.0:1787827928.759760:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827928.759763:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827928.759767:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827928.759773:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827928/real 1787827928] req@ffff9c1a7895f840 x1874668972873216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827928.759776:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972873856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827928.759783:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827928.759790:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827928.759793:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827928.759800:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4379 00000100:00080000:3.0:1787827928.759804:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827928.759807:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827928.759811:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827928.759817:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972873216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827928.759819:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972873984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827928.759872:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827928.759877:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827928.759881:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972873216:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827928.759906:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972873344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827928.759926:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827928.759936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827928/real 1787827928] req@ffff9c1a7895e940 x1874668972873344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827928.759946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972873344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827928.759953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827928.759956:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827928.759959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972873344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827928.759963:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972873472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827928.759984:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827928.759992:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827928/real 1787827928] req@ffff9c1a7895fc00 x1874668972873472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827928.760000:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972873472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827928.760006:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827928.760008:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827928.760010:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972873472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827928.760014:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972873600:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827928.760029:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827928.760037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827928/real 1787827928] req@ffff9c1a7895e580 x1874668972873600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827928.760052:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972873600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827928.760058:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827928.760060:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827928.760063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972873600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827928.760067:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972873728:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827928.760078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827928.760085:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827928/real 1787827928] req@ffff9c1a7895c3c0 x1874668972873728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827928.760101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972873728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827928.760107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827928.760110:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827928.760112:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972873728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827928.760116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972873856:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827928.760126:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827928.760132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827928/real 1787827928] req@ffff9c1a7895cf00 x1874668972873856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827928.760157:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972873856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827928.760163:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827928.760164:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827928.760166:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972873856:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827928.760170:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972873984:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827928.760182:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827928.760187:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827928/real 1787827928] req@ffff9c1a7895f480 x1874668972873984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827928.760195:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972873984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827928.760204:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827928.760207:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827928.760209:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972873984:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827929.271440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827929.271505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827929.527595:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827930.295430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827930.295437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827930.551615:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827931.319424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827931.319434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827931.575550:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827932.343466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827932.343466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827932.599534:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827933.367411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827933.367434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827933.623471:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827933.879461:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827933.879474:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827933.879480:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827933.879493:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4384 00000100:00080000:3.0:1787827933.879507:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827933.879517:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827933.879570:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972874112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827933.879617:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827933.879622:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827933.879625:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827933.879630:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4384 00000100:00080000:3.0:1787827933.879634:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827933.879640:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827933.879654:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827933.879674:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972874240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827933.879690:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972874112:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827933.879719:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827933.879724:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827933.879727:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827933.879734:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4384 00000100:00080000:3.0:1787827933.879739:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827933.879744:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827933.879751:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827933.879767:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972874368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827933.879782:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827933.879789:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1787827933.879791:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827933.879794:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827933.879806:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4384 00000100:00080000:3.0:1787827933.879813:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827933.879819:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827933.879827:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827933.879845:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972874496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827933.879857:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827933.879864:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827933/real 1787827933] req@ffff9c1a7895c000 x1874668972874112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827933.879866:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827933.879873:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827933.879889:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4384 00000100:00080000:3.0:1787827933.879899:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827933.879906:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827933.879913:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827933.879918:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972874112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827933.879933:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972874624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827933.879948:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827933.879952:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827933.879956:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827933.879957:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827933.879971:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4384 00000100:00080000:2.0:1787827933.879977:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827933.879979:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827933.879993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972874112:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827933.879994:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827933.880008:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827933.880024:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972874752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827933.880041:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827933.880048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972874240:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827933.880050:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827933.880057:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827933.880067:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4384 00000100:00080000:3.0:1787827933.880073:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827933.880078:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827933.880085:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827933.880101:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972874880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827933.880117:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827933.880157:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827933/real 1787827933] req@ffff9c1a7895f480 x1874668972874240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827933.880188:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972874240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827933.880203:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827933.880208:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827933.880212:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972874240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827933.880247:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972874368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827933.880280:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827933.880297:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827933/real 1787827933] req@ffff9c1a7895cf00 x1874668972874368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827933.880312:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972874368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827933.880323:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827933.880328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827933.880332:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972874368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827933.880339:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972874496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827933.880405:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827933.880421:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827933/real 1787827933] req@ffff9c1a7895c3c0 x1874668972874496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827933.880436:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972874496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827933.880447:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827933.880451:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827933.880456:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972874496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827933.880463:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972874624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827933.880484:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827933.880495:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827933/real 1787827933] req@ffff9c1a7895e580 x1874668972874624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827933.880509:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972874624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827933.880519:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827933.880522:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827933.880525:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972874624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827933.880565:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972874752:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827933.880589:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827933.880603:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827933/real 1787827933] req@ffff9c1a7895fc00 x1874668972874752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827933.880618:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972874752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827933.880628:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827933.880632:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827933.880636:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972874752:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827933.880643:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972874880:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827933.880661:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827933.880672:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827933/real 1787827933] req@ffff9c1a7895e940 x1874668972874880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827933.880687:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972874880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827933.880697:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827933.880700:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827933.880704:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972874880:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827934.391476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827934.391502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827934.647521:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827935.415465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827935.415477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827935.671582:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827936.439438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827936.439454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827936.695561:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827937.463416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827937.463432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827937.719488:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827938.487485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827938.487528:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827938.743626:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827938.999401:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827938.999425:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827938.999429:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827938.999438:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4389 00000100:00080000:3.0:1787827938.999448:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827938.999454:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827938.999488:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972875008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827938.999514:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827938.999517:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827938.999518:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827938.999531:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4389 00000100:00080000:3.0:1787827938.999533:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827938.999536:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827938.999544:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827938.999553:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972875136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827938.999561:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827938.999564:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827938.999565:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827938.999569:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4389 00000100:00080000:3.0:1787827938.999572:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827938.999574:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827938.999578:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827938.999585:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972875008:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787827938.999588:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972875264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827938.999599:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827938.999601:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827938.999602:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827938.999605:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4389 00000100:00080000:3.0:1787827938.999607:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827938.999610:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827938.999613:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827938.999623:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972875392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827938.999630:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827938.999632:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827938.999634:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827938.999636:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4389 00000100:00080000:3.0:1787827938.999647:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827938.999649:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827938.999653:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827938.999657:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827938.999665:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972875520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827938.999674:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827938.999677:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827938.999678:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827938.999681:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4389 00000100:00080000:3.0:1787827938.999683:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827938.999686:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827938.999689:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827938.999700:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972875648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827938.999707:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827938/real 1787827938] req@ffff9c1a7895f840 x1874668972875008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827938.999709:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827938.999719:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827938.999723:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827938.999729:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4389 00000100:00080000:3.0:1787827938.999732:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827938.999736:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972875008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827938.999737:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827938.999745:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827938.999752:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827938.999757:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972875776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827938.999760:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827938.999768:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972875008:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827938.999778:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972875136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827938.999801:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827938.999811:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827938/real 1787827938] req@ffff9c1a7895f0c0 x1874668972875136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827938.999820:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972875136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827938.999826:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827938.999828:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827938.999830:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972875136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827938.999850:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972875264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827938.999878:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827938.999885:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827938/real 1787827938] req@ffff9c1a7895da40 x1874668972875264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827938.999893:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972875264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827938.999898:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827938.999899:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827938.999901:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972875264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827938.999905:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972875392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827938.999915:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827938.999921:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827938/real 1787827938] req@ffff9c1a7895e1c0 x1874668972875392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827938.999933:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972875392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827938.999939:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827938.999940:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827938.999942:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972875392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827938.999945:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972875520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827938.999974:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827938.999980:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827938/real 1787827938] req@ffff9c1a7895de00 x1874668972875520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827938.999989:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972875520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827938.999995:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827938.999996:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827938.999998:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972875520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827939.000002:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972875648:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827939.000013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827939.000019:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827938/real 1787827939] req@ffff9c1a7895c780 x1874668972875648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827939.000028:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972875648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827939.000038:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827939.000040:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827939.000042:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972875648:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827939.000046:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972875776:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827939.000095:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827939.000102:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827938/real 1787827939] req@ffff9c1a7895cb40 x1874668972875776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827939.000109:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972875776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827939.000116:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827939.000141:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827939.000145:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972875776:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827939.511487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827939.511496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827939.767599:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827940.535423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827940.535426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827940.791522:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827941.559427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827941.559429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827941.815481:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827942.583450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827942.583458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827942.839541:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827943.607487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827943.607493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827943.863555:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827944.119526:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827944.119535:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827944.119539:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827944.119547:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4394 00000100:00080000:0.0:1787827944.119556:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827944.119562:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827944.119604:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972875904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827944.119635:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827944.119637:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827944.119638:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827944.119641:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4394 00000100:00080000:0.0:1787827944.119643:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827944.119646:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827944.119654:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827944.119665:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972876032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827944.119673:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827944.119675:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827944.119677:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827944.119682:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972875904:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827944.119683:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4394 00000100:00080000:0.0:1787827944.119688:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827944.119693:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827944.119696:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827944.119703:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972876160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827944.119711:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827944.119713:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827944.119715:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827944.119717:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4394 00000100:00080000:0.0:1787827944.119721:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827944.119724:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827944.119727:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827944.119734:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972876288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827944.119741:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827944.119744:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827944.119746:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827944.119749:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827944.119752:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4394 00000100:00080000:0.0:1787827944.119754:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827944.119757:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827944.119760:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827944.119768:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972876416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827944.119774:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827944.119777:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827944.119778:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827944.119781:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4394 00000100:00080000:0.0:1787827944.119785:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827944.119785:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827944/real 1787827944] req@ffff9c1a788612c0 x1874668972875904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827944.119792:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827944.119798:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827944.119809:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972876544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827944.119817:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972875904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827944.119837:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827944.119840:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827944.119841:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827944.119843:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827944.119846:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827944.119850:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4394 00000100:00100000:2.0:1787827944.119854:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972875904:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827944.119855:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827944.119863:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827944.119866:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827944.119872:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972876672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827944.119889:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972876032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827944.119908:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827944.119917:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827944/real 1787827944] req@ffff9c1a78863840 x1874668972876032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827944.119925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972876032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827944.119931:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827944.119933:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827944.119935:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972876032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827944.119939:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972876160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827944.119989:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827944.120001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827944/real 1787827944] req@ffff9c1a78860b40 x1874668972876160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827944.120031:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972876160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827944.120040:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827944.120042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827944.120044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972876160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827944.120048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972876288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827944.120059:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827944.120065:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827944/real 1787827944] req@ffff9c1a78862d00 x1874668972876288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827944.120077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972876288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827944.120102:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827944.120105:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827944.120109:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972876288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827944.120116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972876416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827944.120139:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827944.120166:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827944/real 1787827944] req@ffff9c1a78860f00 x1874668972876416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827944.120202:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972876416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827944.120207:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827944.120209:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827944.120212:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972876416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827944.120217:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972876544:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827944.120232:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827944.120239:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827944/real 1787827944] req@ffff9c1a78861a40 x1874668972876544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827944.120247:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972876544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827944.120252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827944.120253:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827944.120255:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972876544:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827944.120258:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972876672:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827944.120268:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827944.120274:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827944/real 1787827944] req@ffff9c1a78862940 x1874668972876672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827944.120282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972876672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827944.120287:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827944.120289:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827944.120290:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972876672:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827944.631491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827944.631495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827944.887597:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827945.655461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827945.655493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827945.911556:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827946.679446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827946.679462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827946.935511:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827947.703481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827947.703492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827947.959611:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827948.727419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827948.727426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827948.983480:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827949.239446:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827949.239454:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827949.239458:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827949.239466:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4399 00000100:00080000:2.0:1787827949.239475:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827949.239481:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827949.239515:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972876800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827949.239537:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827949.239540:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827949.239542:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827949.239544:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4399 00000100:00080000:2.0:1787827949.239547:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827949.239549:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827949.239556:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827949.239565:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972876928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827949.239580:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827949.239584:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827949.239587:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827949.239588:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972876800:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827949.239602:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4399 00000100:00080000:2.0:1787827949.239607:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827949.239610:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827949.239613:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827949.239621:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972877056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827949.239629:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827949.239632:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827949.239633:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827949.239635:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4399 00000100:00080000:2.0:1787827949.239638:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827949.239640:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827949.239644:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827949.239651:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972877184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827949.239657:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827949.239659:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827949.239660:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827949.239662:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4399 00000100:00080000:2.0:1787827949.239665:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827949.239669:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787827949.239672:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787827949.239674:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827949.239683:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972877312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827949.239694:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827949.239696:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827949.239697:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827949.239700:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4399 00000100:00080000:2.0:1787827949.239702:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827949.239705:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827949.239708:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827949.239717:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972877440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827949.239725:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827949/real 1787827949] req@ffff9c1a49d8d680 x1874668972876800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827949.239730:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827949.239739:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827949.239742:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827949.239748:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4399 00000100:00100000:3.0:1787827949.239751:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972876800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827949.239754:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827949.239765:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787827949.239766:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827949.239774:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:2.0:1787827949.239775:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827949.239779:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972876800:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827949.239790:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972877568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827949.239811:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972876928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827949.239837:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827949.239848:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827949/real 1787827949] req@ffff9c1a49d8ed00 x1874668972876928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827949.239857:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972876928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827949.239863:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827949.239865:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827949.239867:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972876928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827949.239871:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972877056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827949.239883:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827949.239889:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827949/real 1787827949] req@ffff9c1a49d8fc00 x1874668972877056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827949.239897:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972877056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827949.239906:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827949.239908:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827949.239910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972877056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827949.239913:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972877184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827949.239923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827949.239929:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827949/real 1787827949] req@ffff9c1a49d8e580 x1874668972877184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827949.239936:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972877184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827949.239941:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827949.239943:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827949.239945:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972877184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827949.239948:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972877312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827949.239958:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827949.239964:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827949/real 1787827949] req@ffff9c1a49d8c3c0 x1874668972877312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827949.239970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972877312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827949.239975:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827949.239977:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827949.239979:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972877312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827949.239982:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972877440:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827949.239992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827949.239998:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827949/real 1787827949] req@ffff9c1a49d8e1c0 x1874668972877440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827949.240004:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972877440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827949.240009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827949.240011:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827949.240012:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972877440:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827949.240016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972877568:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827949.240025:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827949.240031:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827949/real 1787827949] req@ffff9c1a49d8cb40 x1874668972877568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827949.240038:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972877568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827949.240043:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827949.240045:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827949.240046:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972877568:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827949.751442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827949.751443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827950.007470:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827950.775477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827950.775477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827951.031544:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827951.799442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827951.799450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827952.055464:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827952.823493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827952.823504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827953.079525:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827953.847510:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827953.847510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827954.103586:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827954.359505:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827954.359516:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827954.359521:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827954.359533:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4404 00000100:00080000:2.0:1787827954.359546:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827954.359555:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787827954.359601:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972877696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827954.359644:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827954.359651:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827954.359654:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827954.359661:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4404 00000100:00080000:2.0:1787827954.359667:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827954.359677:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827954.359693:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827954.359709:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972877696:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827954.359721:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972877824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827954.359738:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827954.359743:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827954.359746:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827954.359751:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4404 00000100:00080000:2.0:1787827954.359772:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827954.359778:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787827954.359785:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787827954.359791:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827954.359830:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972877952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827954.359844:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827954.359850:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827954.359854:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827954.359862:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4404 00000100:00080000:2.0:1787827954.359870:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827954.359878:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827954.359887:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827954.359909:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972878080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827954.359926:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827954.359933:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827954.359937:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827954.359946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827954/real 1787827954] req@ffff9c1a49d8cf00 x1874668972877696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827954.359947:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4404 00000100:00080000:2.0:1787827954.359959:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827954.359969:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827954.359978:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827954.359982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972877696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827954.359999:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972878208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827954.359999:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827954.360012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827954.360024:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972877696:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787827954.360027:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827954.360033:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827954.360037:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827954.360043:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4404 00000100:00080000:2.0:1787827954.360047:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827954.360051:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827954.360057:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827954.360070:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972878336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827954.360102:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827954.360106:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827954.360108:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827954.360112:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4404 00000100:00080000:2.0:1787827954.360116:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827954.360121:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787827954.360126:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827954.360139:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972878464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827954.360207:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972877824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827954.360243:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827954.360264:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827954/real 1787827954] req@ffff9c1a49d8da40 x1874668972877824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827954.360418:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972877824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827954.360449:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827954.360455:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827954.360462:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972877824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827954.360473:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972877952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827954.360508:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827954.360543:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827954/real 1787827954] req@ffff9c1a49d8c000 x1874668972877952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827954.360576:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972877952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827954.360591:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827954.360597:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827954.360603:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972877952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827954.360614:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972878080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827954.360662:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827954.360678:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827954/real 1787827954] req@ffff9c1a49d8de00 x1874668972878080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827954.360692:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972878080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827954.360702:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827954.360706:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827954.360709:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972878080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827954.360716:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972878208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827954.360734:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827954.360744:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827954/real 1787827954] req@ffff9c1a49d8c780 x1874668972878208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827954.360757:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972878208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827954.360774:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827954.360778:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827954.360798:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972878208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827954.360819:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972878336:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827954.360838:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827954.360847:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827954/real 1787827954] req@ffff9c1a49d8f0c0 x1874668972878336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827954.360858:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972878336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827954.360866:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827954.360870:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827954.360873:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972878336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827954.360878:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972878464:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827954.360892:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827954.360900:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827954/real 1787827954] req@ffff9c1a49d8cf00 x1874668972878464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827954.360911:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972878464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827954.360919:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827954.360921:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827954.360925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972878464:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827954.871462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827954.871546:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827955.127583:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827955.895472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827955.895500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827956.151502:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827956.919437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827956.919461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827957.175498:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827957.943488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827957.943489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827958.199596:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827958.967521:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827958.967522:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827959.223570:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827959.479490:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827959.479499:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827959.479504:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827959.479512:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4409 00000100:00080000:1.0:1787827959.479522:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827959.479529:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827959.479567:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972878592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827959.479592:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827959.479595:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827959.479597:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827959.479600:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4409 00000100:00080000:1.0:1787827959.479603:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827959.479606:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827959.479616:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827959.479631:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972878720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827959.479632:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972878592:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827959.479648:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827959.479651:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827959.479652:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827959.479655:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4409 00000100:00080000:1.0:1787827959.479658:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827959.479660:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827959.479664:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827959.479675:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972878848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827959.479683:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827959.479685:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827959.479687:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827959.479690:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4409 00000100:00080000:1.0:1787827959.479693:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827959.479695:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787827959.479701:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:1.0:1787827959.479702:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827959.479714:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972878976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827959.479722:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827959.479725:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827959.479726:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827959.479729:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4409 00000100:00080000:1.0:1787827959.479731:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827959.479734:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827959.479738:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827959.479752:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972879104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827959.479753:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827959/real 1787827959] req@ffff9c1a78a78f00 x1874668972878592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827959.479767:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827959.479775:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827959.479778:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827959.479784:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4409 00000100:00080000:1.0:1787827959.479787:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827959.479790:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827959.479794:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827959.479799:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972878592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827959.479805:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972879232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827959.479820:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827959.479821:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827959.479828:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827959.479831:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827959.479832:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827959.479838:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972878592:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827959.479839:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4409 00000100:00080000:1.0:1787827959.479858:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827959.479862:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827959.479866:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827959.479868:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972878720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827959.479878:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972879360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827959.479896:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827959.479910:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827959/real 1787827959] req@ffff9c1a78a7a580 x1874668972878720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827959.479921:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972878720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827959.479928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827959.479930:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827959.479933:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972878720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827959.479937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972878848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827959.479950:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827959.479957:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827959/real 1787827959] req@ffff9c1a78a78000 x1874668972878848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827959.479966:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972878848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827959.479971:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827959.479974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827959.479976:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972878848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827959.479980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972878976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827959.479991:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827959.479998:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827959/real 1787827959] req@ffff9c1a78a7b480 x1874668972878976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827959.480006:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972878976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827959.480012:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827959.480014:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827959.480016:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972878976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827959.480020:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972879104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827959.480031:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827959.480037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827959/real 1787827959] req@ffff9c1a78a792c0 x1874668972879104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827959.480072:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972879104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827959.480079:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827959.480081:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827959.480084:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972879104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827959.480088:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972879232:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827959.480105:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827959.480113:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827959/real 1787827959] req@ffff9c1a78a7b840 x1874668972879232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827959.480122:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972879232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827959.480128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827959.480130:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827959.480133:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972879232:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827959.480147:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972879360:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827959.480158:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827959.480165:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827959/real 1787827959] req@ffff9c1a78a79a40 x1874668972879360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827959.480173:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972879360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827959.480184:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827959.480186:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827959.480189:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972879360:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827959.991508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827959.991509:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827960.247622:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827961.015444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827961.015470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827961.271583:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827962.039466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827962.039505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827962.295583:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827963.063519:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827963.063530:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827963.319601:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827964.087503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827964.087505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827964.343492:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827964.599440:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827964.599451:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827964.599454:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827964.599462:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4414 00000100:00080000:1.0:1787827964.599471:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827964.599478:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827964.599522:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972879488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827964.599544:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827964.599547:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827964.599548:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827964.599551:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4414 00000100:00080000:1.0:1787827964.599553:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827964.599555:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827964.599563:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827964.599573:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972879616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827964.599583:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972879488:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827964.599585:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827964.599592:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827964.599594:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827964.599599:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4414 00000100:00080000:1.0:1787827964.599603:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827964.599608:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827964.599614:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827964.599627:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972879744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827964.599638:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787827964.599640:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827964.599642:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827964.599645:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827964.599648:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4414 00000100:00080000:1.0:1787827964.599653:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827964.599656:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827964.599660:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827964.599669:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972879872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827964.599676:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827964.599679:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827964/real 1787827964] req@ffff9c1a78a79a40 x1874668972879488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827964.599680:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827964.599683:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827964.599690:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4414 00000100:00080000:1.0:1787827964.599694:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827964.599698:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827964.599702:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827964.599704:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972879488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827964.599712:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972880000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827964.599715:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827964.599723:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827964.599724:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827964.599730:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787827964.599731:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972879488:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787827964.599732:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827964.599740:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4414 00000100:00080000:1.0:1787827964.599743:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827964.599745:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827964.599749:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827964.599755:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972880128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827964.599762:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972879616:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827964.599776:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827964.599779:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827964.599780:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827964.599783:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4414 00000100:00080000:1.0:1787827964.599786:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827964.599789:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827964.599792:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827964.599799:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827964.599802:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972880256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827964.599816:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827964/real 1787827964] req@ffff9c1a78a7b840 x1874668972879616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827964.599828:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972879616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827964.599835:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827964.599837:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827964.599839:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972879616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827964.599844:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972879744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827964.599879:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827964.599898:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827964/real 1787827964] req@ffff9c1a78a792c0 x1874668972879744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827964.599922:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972879744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827964.599938:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827964.599944:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827964.599966:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972879744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827964.599979:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972879872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827964.600050:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827964.600080:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827964/real 1787827964] req@ffff9c1a78a7b480 x1874668972879872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827964.600094:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972879872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827964.600101:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827964.600103:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827964.600105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972879872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827964.600109:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972880000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787827964.600124:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827964.600131:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827964/real 1787827964] req@ffff9c1a78a78000 x1874668972880000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827964.600139:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972880000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827964.600144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827964.600146:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827964.600148:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972880000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787827964.600160:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972880128:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827964.600198:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827964.600209:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827964/real 1787827964] req@ffff9c1a78a7a580 x1874668972880128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827964.600223:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972880128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827964.600249:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827964.600253:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827964.600256:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972880128:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827964.600263:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972880256:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787827964.600282:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787827964.600309:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827964/real 1787827964] req@ffff9c1a78a78f00 x1874668972880256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827964.600319:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972880256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827964.600337:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827964.600340:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787827964.600377:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972880256:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827965.111465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827965.111465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827965.367594:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827966.135464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827966.135483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827966.391565:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827967.159495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827967.159496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827967.415606:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827968.183416:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827968.183425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827968.439636:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827969.207510:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827969.207545:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827969.463578:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827969.719545:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827969.719557:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827969.719562:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827969.719584:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4420 00000100:00080000:3.0:1787827969.719598:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827969.719607:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827969.719656:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972880384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827969.719708:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827969.719715:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827969.719719:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827969.719726:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4420 00000100:00080000:3.0:1787827969.719733:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827969.719741:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827969.719770:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827969.719799:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972880512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827969.719815:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972880384:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827969.719854:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827969.719860:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827969.719863:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827969.719870:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4420 00000100:00080000:3.0:1787827969.719876:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827969.719882:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827969.719899:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827969.719917:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972880640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827969.719932:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827969.719936:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827969.719939:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827969.719944:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4420 00000100:00080000:3.0:1787827969.719949:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827969.719953:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827969.719959:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827969.720009:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972880768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827969.720023:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827969.720027:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827969.720031:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827969.720038:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4420 00000100:00080000:3.0:1787827969.720044:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827969.720051:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972880512:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827969.720056:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827969.720071:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827969.720095:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972880896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827969.720110:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827969.720115:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827969.720118:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827969.720124:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4420 00000100:00080000:3.0:1787827969.720130:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827969.720134:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972880640:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827969.720141:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827969.720153:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827969.720172:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972881024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827969.720185:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827969.720190:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827969.720194:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827969.720199:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972880768:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787827969.720203:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4420 00000100:00080000:3.0:1787827969.720212:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827969.720220:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827969.720233:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827969.720250:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972881152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827969.720311:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972880896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827969.720387:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972881024:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827969.720431:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972881152:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827970.231479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827970.231480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827970.487558:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827971.255530:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827971.255530:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827971.511665:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827972.279492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827972.279508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827972.535584:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827972.791703:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787827972.791759:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827972.791779:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827972.791833:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827972.791850:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827972.791865:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827972.791881:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827972.791898:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787827972.791917:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787827972.791943:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827969/real 1787827972] req@ffff9c1a7895e1c0 x1874668972880896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827972.792015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972880896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827972.792038:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827972.792048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827972.792057:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972880896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827972.792071:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827969/real 1787827972] req@ffff9c1a7895da40 x1874668972881024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827972.792097:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972881024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827972.792115:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827972.792122:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827972.792129:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972881024:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827972.792142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827969/real 1787827972] req@ffff9c1a7895f0c0 x1874668972881152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827972.792167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972881152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827972.792184:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827972.792190:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827972.792197:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972881152:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827972.792210:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827969/real 1787827972] req@ffff9c1a7895cb40 x1874668972880512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827972.792234:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972880512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827972.792251:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827972.792258:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827972.792278:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972880512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827972.792291:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827969/real 1787827972] req@ffff9c1a7895c780 x1874668972880640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827972.792316:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972880640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827972.792333:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827972.792340:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827972.792396:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972880640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827972.792420:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827969/real 1787827972] req@ffff9c1a7895de00 x1874668972880768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827972.792448:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972880768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827972.792465:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827972.792471:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827972.792477:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972880768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827972.792490:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827969/real 1787827972] req@ffff9c1a7895d2c0 x1874668972880384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827972.792516:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972880384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827972.792531:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827972.792539:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827972.792546:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972880384:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787827973.303549:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827973.303570:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827973.559552:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827974.327462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827974.327468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827974.583552:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827974.839542:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827974.839557:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827974.839563:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827974.839577:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4425 00000100:00080000:1.0:1787827974.839592:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827974.839601:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827974.839652:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972881280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827974.839691:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827974.839699:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827974.839702:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827974.839711:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4425 00000100:00080000:1.0:1787827974.839718:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827974.839726:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827974.839744:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827974.839768:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972881280:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827974.839773:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972881408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827974.839800:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827974.839807:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827974.839810:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827974.839818:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4425 00000100:00080000:1.0:1787827974.839827:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827974.839835:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827974.839845:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827974.839856:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827974.839872:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972881536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827974.839897:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827974.839903:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827974/real 1787827974] req@ffff9c1a78a7a940 x1874668972881280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827974.839906:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827974.839912:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827974.839929:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4425 00000100:00080000:1.0:1787827974.839962:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827974.839969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972881280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827974.839977:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827974.839998:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827974.840002:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827974.840013:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827974.840018:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972881280:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787827974.840029:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972881664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827974.840048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972881408:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827974.840056:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827974.840065:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827974.840069:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827974.840079:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4425 00000800:00000100:2.0:1787827974.840089:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787827974.840090:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827974.840103:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827974.840115:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827974.840120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827974/real 1787827974] req@ffff9c1a78a783c0 x1874668972881408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827974.840147:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972881792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827974.840150:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972881408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827974.840210:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827974.840211:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827974.840226:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787827974.840227:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827974.840235:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827974.840241:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972881408:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827974.840249:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4425 00000100:00080000:1.0:1787827974.840262:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827974.840271:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972881536:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827974.840278:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827974.840293:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827974.840318:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827974.840321:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972881920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827974.840341:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827974/real 1787827974] req@ffff9c1a78a7a1c0 x1874668972881536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827974.840429:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972881536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827974.840439:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827974.840443:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827974.840445:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827974.840460:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972881536:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827974.840461:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827974.840470:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827974.840484:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4425 00000100:00080000:1.0:1787827974.840494:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827974.840498:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972881664:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787827974.840509:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787827974.840524:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827974.840550:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827974.840556:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972882048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827974.840576:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827974/real 1787827974] req@ffff9c1a78a78780 x1874668972881664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827974.840601:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972881664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827974.840611:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827974.840616:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827974.840621:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972881664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827974.840631:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972881792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827974.840654:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827974.840666:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827974/real 1787827974] req@ffff9c1a78a7bc00 x1874668972881792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827974.840677:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972881792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827974.840702:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827974.840707:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827974.840710:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972881792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827974.840717:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972881920:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827974.840742:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827974.840760:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827974/real 1787827974] req@ffff9c1a78a7b0c0 x1874668972881920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827974.840773:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972881920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827974.840785:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827974.840788:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827974.840793:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972881920:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827974.840824:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972882048:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827974.840850:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827974.840864:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827974/real 1787827974] req@ffff9c1a78a79e00 x1874668972882048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827974.840881:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972882048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827974.840890:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827974.840893:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827974.840897:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972882048:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787827975.351464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827975.351464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827975.607555:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827976.375465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827976.375494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827976.631521:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827977.399468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787827977.399474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827977.655516:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827978.423432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827978.423489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827978.679723:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827979.447469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827979.447614:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827979.703521:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827979.959548:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827979.959558:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827979.959562:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827979.959571:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4430 00000100:00080000:0.0:1787827979.959581:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827979.959588:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827979.959628:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972882176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827979.959664:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827979.959675:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827979.959678:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827979.959684:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4430 00000100:00080000:0.0:1787827979.959688:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827979.959693:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827979.959703:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827979.959717:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972882304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827979.959726:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972882176:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827979.959736:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827979.959741:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827979.959744:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827979.959751:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4430 00000100:00080000:0.0:1787827979.959756:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827979.959763:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827979.959772:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827979.959788:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972882432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827979.959795:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827979.959807:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827979.959811:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827979.959813:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827979.959818:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4430 00000100:00080000:0.0:1787827979.959821:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827979.959825:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827979.959829:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827979.959838:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827979/real 1787827979] req@ffff9c1a788621c0 x1874668972882176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827979.959842:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972882560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827979.959872:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827979.959875:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827979.959878:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827979.959880:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972882176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827979.959884:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4430 00000100:00080000:0.0:1787827979.959890:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827979.959896:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827979.959920:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827979.959922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827979.959929:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787827979.959951:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972882688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827979.959953:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972882176:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827979.959978:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827979.959980:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827979.959982:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827979.959987:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4430 00000100:00080000:0.0:1787827979.959991:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827979.959992:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972882304:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787827979.959999:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827979.960006:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827979.960018:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972882816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827979.960029:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827979.960029:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827979.960036:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827979.960039:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827979.960044:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4430 00000100:00080000:0.0:1787827979.960048:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827979.960051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827979/real 1787827979] req@ffff9c1a78860780 x1874668972882304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827979.960055:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827979.960063:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827979.960077:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972882944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827979.960102:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972882304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827979.960117:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827979.960122:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827979.960129:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972882304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827979.960139:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972882432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827979.960162:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827979.960173:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827979/real 1787827979] req@ffff9c1a78861e00 x1874668972882432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827979.960182:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972882432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827979.960189:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827979.960192:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827979.960194:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972882432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827979.960199:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972882560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827979.960212:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827979.960218:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827979/real 1787827979] req@ffff9c1a78862580 x1874668972882560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827979.960227:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972882560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827979.960234:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827979.960236:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827979.960238:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972882560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827979.960242:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972882688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827979.960254:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827979.960261:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827979/real 1787827979] req@ffff9c1a78861680 x1874668972882688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827979.960269:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972882688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827979.960275:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827979.960277:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827979.960279:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972882688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827979.960293:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972882816:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827979.960317:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827979.960325:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827979/real 1787827979] req@ffff9c1a788630c0 x1874668972882816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827979.960334:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972882816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827979.960340:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827979.960373:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827979.960376:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972882816:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827979.960381:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972882944:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827979.960396:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827979.960404:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827979/real 1787827979] req@ffff9c1a78863c00 x1874668972882944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827979.960418:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972882944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827979.960425:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827979.960427:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827979.960430:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972882944:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827980.471507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827980.471549:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827980.727560:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827981.495481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827981.495537:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827981.751559:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827982.519478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787827982.519551:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827982.775605:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827983.543471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827983.543544:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827983.799537:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827984.567452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827984.567503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827984.823517:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827985.079453:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827985.079464:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827985.079470:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827985.079483:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4435 00000100:00080000:0.0:1787827985.079497:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827985.079506:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827985.079553:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972883072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827985.079590:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827985.079595:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827985.079598:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827985.079603:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4435 00000100:00080000:0.0:1787827985.079607:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827985.079611:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827985.079623:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827985.079639:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972883200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827985.079651:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827985.079655:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827985.079657:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827985.079661:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4435 00000100:00080000:0.0:1787827985.079666:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827985.079670:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827985.079675:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827985.079687:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972883328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827985.079712:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827985.079719:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787827985.079719:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972883072:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827985.079727:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827985.079741:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4435 00000100:00080000:0.0:1787827985.079745:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827985.079749:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827985.079756:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827985.079771:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972883456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827985.079783:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827985.079788:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827985.079791:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827985.079795:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4435 00000100:00080000:0.0:1787827985.079800:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827985.079805:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827985.079810:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827985.079820:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787827985.079827:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972883584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827985.079842:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827985.079847:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827985.079850:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827985.079855:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4435 00000100:00080000:0.0:1787827985.079861:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827985.079891:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827985.079899:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827985.079909:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827985/real 1787827985] req@ffff9c1a78860000 x1874668972883072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827985.079928:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972883712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827985.079946:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827985.079950:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827985.079953:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827985.079962:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972883072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827985.079964:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4435 00000100:00080000:0.0:1787827985.079973:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827985.079983:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827985.079994:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827985.080017:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827985.080018:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972883840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827985.080040:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827985.080054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972883072:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787827985.080068:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972883200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827985.080107:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827985.080127:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827985/real 1787827985] req@ffff9c1a78863c00 x1874668972883200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827985.080144:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972883200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827985.080155:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827985.080159:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827985.080163:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972883200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827985.080193:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972883328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827985.080215:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827985.080227:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827985/real 1787827985] req@ffff9c1a788630c0 x1874668972883328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827985.080241:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972883328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827985.080251:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827985.080255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827985.080259:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972883328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827985.080266:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972883456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827985.080284:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827985.080295:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827985/real 1787827985] req@ffff9c1a78861680 x1874668972883456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827985.080308:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972883456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827985.080320:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827985.080323:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827985.080327:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972883456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827985.080333:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972883584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787827985.080397:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827985.080412:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827985/real 1787827985] req@ffff9c1a78862580 x1874668972883584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827985.080427:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972883584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827985.080437:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827985.080441:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827985.080445:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972883584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787827985.080452:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972883712:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827985.080475:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827985.080486:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827985/real 1787827985] req@ffff9c1a78861e00 x1874668972883712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827985.080500:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972883712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827985.080510:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827985.080513:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827985.080518:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972883712:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787827985.080525:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972883840:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787827985.080543:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787827985.080553:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827985/real 1787827985] req@ffff9c1a78860780 x1874668972883840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827985.080566:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972883840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827985.080577:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827985.080580:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787827985.080584:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972883840:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827985.591458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827985.591488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827985.847502:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827986.615477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827986.615506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827986.871544:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827987.639469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827987.639515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827987.895581:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827988.663457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827988.663480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827988.919580:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827989.687444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827989.687474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827989.943610:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827990.199488:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827990.199502:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827990.199508:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827990.199520:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4440 00000100:00080000:0.0:1787827990.199537:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827990.199547:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787827990.199612:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972883968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827990.199655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827990.199662:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827990.199665:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827990.199672:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4440 00000100:00080000:0.0:1787827990.199678:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827990.199684:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827990.199698:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827990.199722:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972884096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827990.199731:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972883968:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827990.199749:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827990.199756:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827990.199759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827990.199767:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4440 00000100:00080000:0.0:1787827990.199774:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827990.199779:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827990.199788:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827990.199807:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972884224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827990.199853:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827990.199863:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827990.199869:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827990.199877:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827990.199888:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4440 00000100:00080000:0.0:1787827990.199899:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827990.199907:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827990.199921:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827990.199948:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972884352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827990.199956:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827990/real 1787827990] req@ffff9c1a788621c0 x1874668972883968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827990.199997:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827990.200007:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827990.200014:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972883968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827990.200015:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827990.200036:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4440 00000100:00080000:1.0:1787827990.200042:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827990.200052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787827990.200059:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827990.200059:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972883968:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787827990.200079:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827990.200093:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827990.200108:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972884096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827990.200127:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972884480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827990.200151:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827990.200155:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827990.200166:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827990.200171:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827990.200176:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827990/real 1787827990] req@ffff9c1a78862940 x1874668972884096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827990.200186:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4440 00000100:00080000:0.0:1787827990.200206:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827990.200211:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972884096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827990.200225:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787827990.200232:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827990.200239:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827990.200243:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827990.200254:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972884096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827990.200264:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972884224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787827990.200272:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972884608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827990.200290:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827990.200295:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827990.200305:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827990.200305:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787827990.200322:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4440 00000100:00080000:0.0:1787827990.200331:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827990.200339:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827990/real 1787827990] req@ffff9c1a78861a40 x1874668972884224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827990.200382:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787827990.200395:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827990.200416:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972884224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827990.200428:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972884736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827990.200433:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827990.200450:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827990.200463:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972884224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827990.200482:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972884352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827990.200515:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827990.200548:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827990/real 1787827990] req@ffff9c1a78860f00 x1874668972884352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827990.200574:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972884352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827990.200588:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827990.200594:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827990.200599:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972884352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827990.200628:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972884480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827990.200649:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827990.200662:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827990/real 1787827990] req@ffff9c1a78862d00 x1874668972884480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827990.200676:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972884480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827990.200687:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827990.200693:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827990.200699:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972884480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827990.200711:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972884608:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827990.200748:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827990.200770:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827990/real 1787827990] req@ffff9c1a78860b40 x1874668972884608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827990.200796:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972884608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827990.200815:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827990.200819:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827990.200824:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972884608:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827990.200862:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972884736:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827990.200899:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827990.200921:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827990/real 1787827990] req@ffff9c1a78863840 x1874668972884736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827990.200946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972884736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827990.200964:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827990.200971:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827990.200978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972884736:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827990.711458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827990.711477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827990.967568:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827991.735454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827991.735459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827991.991577:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827992.759504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827992.759637:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827993.015637:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827993.783444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827993.783489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827994.039552:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827994.807496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827994.807497:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787827995.063605:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827995.319489:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827995.319502:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827995.319508:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827995.319522:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4445 00000100:00080000:3.0:1787827995.319536:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827995.319545:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787827995.319606:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972884864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827995.319670:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827995.319676:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827995.319678:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827995.319684:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4445 00000100:00080000:3.0:1787827995.319689:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827995.319695:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827995.319709:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827995.319727:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972884992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827995.319743:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972884864:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827995.319747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827995.319760:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827995.319763:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827995.319769:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4445 00000100:00080000:3.0:1787827995.319773:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827995.319778:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827995.319784:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827995.319828:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972885120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827995.319844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827995.319861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827995.319864:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827995.319869:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4445 00000100:00080000:3.0:1787827995.319875:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827995.319879:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827995.319888:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827995.319899:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827995.319917:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972885248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827995.319931:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827995.319936:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827995.319939:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827995.319947:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827995/real 1787827995] req@ffff9c1a7895f840 x1874668972884864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827995.319950:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4445 00000100:00080000:3.0:1787827995.319958:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827995.319971:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827995.319980:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827995.319998:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972885376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827995.320010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972884864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827995.320030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827995.320037:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827995.320041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827995.320047:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787827995.320052:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827995.320059:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972884864:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787827995.320070:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4445 00000100:00080000:3.0:1787827995.320081:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827995.320091:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787827995.320099:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972884992:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787827995.320106:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827995.320134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972885504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827995.320149:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827995.320153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827995.320156:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827995.320162:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4445 00000100:00080000:3.0:1787827995.320169:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827995.320176:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787827995.320181:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787827995.320205:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827995.320218:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827995/real 1787827995] req@ffff9c1a7895e940 x1874668972884992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827995.320225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972885632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827995.320269:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972884992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827995.320285:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827995.320291:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827995.320301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972884992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827995.320316:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972885120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827995.320408:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827995.320428:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827995/real 1787827995] req@ffff9c1a7895fc00 x1874668972885120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827995.320444:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972885120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827995.320456:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827995.320460:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827995.320464:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972885120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827995.320472:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972885248:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827995.320497:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827995.320526:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827995/real 1787827995] req@ffff9c1a7895e580 x1874668972885248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827995.320542:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972885248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827995.320553:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827995.320556:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827995.320560:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972885248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827995.320568:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972885376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787827995.320593:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827995.320614:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827995/real 1787827995] req@ffff9c1a7895c3c0 x1874668972885376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827995.320647:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972885376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827995.320665:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827995.320672:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827995.320679:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972885376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787827995.320708:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972885504:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827995.320739:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827995.320753:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827995/real 1787827995] req@ffff9c1a7895cf00 x1874668972885504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827995.320768:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972885504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827995.320779:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827995.320783:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827995.320787:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972885504:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787827995.320822:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972885632:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787827995.320860:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787827995.320875:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827995/real 1787827995] req@ffff9c1a7895f480 x1874668972885632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827995.320890:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972885632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827995.320901:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827995.320904:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787827995.320908:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972885632:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787827995.831487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827995.831491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827996.087658:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827996.855469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827996.855475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787827997.111577:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827997.879481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827997.879554:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787827998.135620:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827998.903450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787827998.903454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787827999.159453:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827999.927514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787827999.927514:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828000.183604:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828000.439436:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828000.439446:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828000.439450:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828000.439459:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4450 00000100:00080000:0.0:1787828000.439469:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828000.439475:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828000.439527:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972885760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828000.439559:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828000.439562:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828000.439564:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828000.439568:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4450 00000100:00080000:0.0:1787828000.439572:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828000.439575:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828000.439584:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828000.439605:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972885888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828000.439614:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828000.439617:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828000.439620:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828000.439623:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4450 00000100:00080000:0.0:1787828000.439626:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828000.439629:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828000.439633:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828000.439643:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972885760:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828000.439644:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972886016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828000.439660:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828000.439664:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828000.439665:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828000.439669:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4450 00000100:00080000:0.0:1787828000.439671:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828000.439674:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828000.439677:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828000.439688:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972886144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828000.439694:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828000.439696:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828000.439697:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828000.439701:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4450 00000100:00080000:0.0:1787828000.439705:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828000.439707:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828000.439712:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828000.439717:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828000.439726:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972886272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828000.439735:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828000.439737:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828000.439739:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828000.439741:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4450 00000100:00080000:0.0:1787828000.439745:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828000.439748:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828000.439752:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828000.439780:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972886400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828000.439780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828000/real 1787828000] req@ffff9c1a788612c0 x1874668972885760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828000.439812:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828000.439816:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828000.439817:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828000.439821:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4450 00000100:00080000:0.0:1787828000.439825:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828000.439828:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972885760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828000.439832:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828000.439840:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828000.439846:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828000.439853:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972886528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828000.439856:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828000.439865:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972885760:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828000.439878:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972885888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828000.439902:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828000.439913:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828000/real 1787828000] req@ffff9c1a78863840 x1874668972885888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828000.439922:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972885888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828000.439929:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828000.439931:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828000.439934:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972885888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828000.439954:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972886016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828000.439967:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828000.439974:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828000/real 1787828000] req@ffff9c1a78860b40 x1874668972886016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828000.439982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972886016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828000.439989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828000.439991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828000.439993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972886016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828000.439997:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972886144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828000.440009:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828000.440015:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828000/real 1787828000] req@ffff9c1a78862d00 x1874668972886144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828000.440024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972886144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828000.440030:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828000.440032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828000.440034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972886144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828000.440038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972886272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828000.440049:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828000.440055:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828000/real 1787828000] req@ffff9c1a78860f00 x1874668972886272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828000.440063:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972886272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828000.440069:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828000.440071:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828000.440074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972886272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828000.440078:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972886400:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828000.440089:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828000.440095:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828000/real 1787828000] req@ffff9c1a78861a40 x1874668972886400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828000.440110:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972886400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828000.440116:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828000.440118:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828000.440120:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972886400:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828000.440123:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972886528:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828000.440134:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828000.440141:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828000/real 1787828000] req@ffff9c1a78862940 x1874668972886528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828000.440160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972886528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828000.440167:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828000.440170:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828000.440172:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972886528:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828000.951410:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828000.951417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828001.207520:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828001.975458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828001.975495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828002.231642:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828002.999469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828002.999544:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828003.255611:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828004.023404:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828004.023425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828004.279477:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828005.047442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828005.047453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828005.303490:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828005.559499:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828005.559512:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828005.559535:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828005.559553:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4455 00000100:00080000:2.0:1787828005.559583:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828005.559595:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828005.559659:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972886656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828005.559698:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828005.559703:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828005.559706:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828005.559711:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4455 00000100:00080000:2.0:1787828005.559717:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828005.559767:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828005.559785:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828005.559793:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972886656:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828005.559802:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972886784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828005.559814:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828005.559817:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828005.559820:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828005.559824:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4455 00000100:00080000:2.0:1787828005.559829:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828005.559833:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828005.559837:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828005.559847:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972886912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828005.559857:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828005.559863:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828005.559864:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828005.559870:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828005.559875:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4455 00000100:00080000:2.0:1787828005.559880:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828005.559884:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828005.559888:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828005.559899:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972887040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828005.559908:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828005.559912:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828005/real 1787828005] req@ffff9c1a49d8cb40 x1874668972886656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828005.559913:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828005.559917:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828005.559927:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4455 00000100:00080000:2.0:1787828005.559935:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828005.559946:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828005.559953:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828005.559954:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972886656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828005.559970:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828005.559976:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972887168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828005.559977:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828005.560008:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972886656:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828005.560043:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972886784:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828005.560045:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828005.560059:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828005.560064:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828005.560075:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4455 00000100:00080000:2.0:1787828005.560085:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828005.560087:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828005.560110:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828005.560120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828005/real 1787828005] req@ffff9c1a49d8e1c0 x1874668972886784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828005.560122:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828005.560160:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972887296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828005.560165:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972886784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828005.560203:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828005.560213:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828005.560214:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828005.560226:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828005.560233:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787828005.560242:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4455 00000100:00100000:1.0:1787828005.560251:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972886784:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828005.560253:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828005.560268:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828005.560275:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972886912:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787828005.560281:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828005.560309:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972887424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828005.560324:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828005.560349:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828005/real 1787828005] req@ffff9c1a49d8c3c0 x1874668972886912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828005.560379:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972886912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828005.560461:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828005.560467:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828005.560488:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972886912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828005.560495:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972887040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828005.560518:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828005.560528:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828005/real 1787828005] req@ffff9c1a49d8e580 x1874668972887040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828005.560539:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972887040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828005.560590:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828005.560595:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828005.560599:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972887040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828005.560633:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972887168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828005.560658:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828005.560672:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828005/real 1787828005] req@ffff9c1a49d8fc00 x1874668972887168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828005.560686:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972887168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828005.560697:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828005.560701:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828005.560705:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972887168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828005.560711:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972887296:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828005.560775:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828005.560786:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828005/real 1787828005] req@ffff9c1a49d8ed00 x1874668972887296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828005.560797:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972887296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828005.560804:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828005.560807:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828005.560809:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972887296:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828005.560814:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972887424:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828005.560858:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828005.560870:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828005/real 1787828005] req@ffff9c1a49d8d680 x1874668972887424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828005.560884:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972887424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828005.560900:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828005.560903:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828005.560907:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972887424:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828006.071474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828006.071579:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828006.327528:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828007.095495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828007.095534:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828007.351551:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828008.119479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828008.119545:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828008.375583:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828009.143470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828009.143470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828009.399532:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828010.167461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828010.167574:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828010.423629:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828010.679496:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828010.679525:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828010.679530:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828010.679540:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4461 00000100:00080000:0.0:1787828010.679551:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828010.679559:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828010.679637:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972887552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828010.679677:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828010.679682:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828010.679704:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828010.679712:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4461 00000100:00080000:0.0:1787828010.679717:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828010.679724:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828010.679749:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828010.679770:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972887680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828010.679786:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972887552:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828010.679792:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828010.679803:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828010.679809:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828010.679817:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4461 00000100:00080000:0.0:1787828010.679821:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828010.679826:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828010.679832:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828010.679861:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972887808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828010.679868:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828010.679871:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828010.679872:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828010.679875:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4461 00000100:00080000:0.0:1787828010.679885:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828010.679889:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828010.679892:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828010.679902:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972887936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828010.679902:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828010.679911:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828010.679914:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828010.679915:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828010.679918:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4461 00000100:00080000:0.0:1787828010.679921:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828010.679923:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828010.679926:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828010.679963:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972888064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828010.679976:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828010.679981:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828010.679984:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828010.679994:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4461 00000100:00100000:1.0:1787828010.679999:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828010/real 1787828010] req@ffff9c1a788621c0 x1874668972887552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828010.680000:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828010.680009:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828010.680018:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828010.680053:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972888192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828010.680064:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828010.680069:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828010.680074:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828010.680075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972887552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828010.680081:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4461 00000100:00080000:0.0:1787828010.680086:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828010.680097:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828010.680106:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828010.680121:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972888320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828010.680126:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828010.680143:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828010.680164:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972887552:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828010.680204:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972887680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828010.680260:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828010.680274:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828010/real 1787828010] req@ffff9c1a78860780 x1874668972887680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828010.680298:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972887680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828010.680308:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828010.680330:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828010.680338:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972887680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828010.680350:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972887808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828010.680396:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828010.680418:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828010/real 1787828010] req@ffff9c1a78861e00 x1874668972887808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828010.680500:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972887808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828010.680537:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828010.680540:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828010.680544:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972887808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828010.680551:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972887936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828010.680572:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828010.680617:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828010/real 1787828010] req@ffff9c1a78862580 x1874668972887936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828010.680654:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972887936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828010.680671:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828010.680678:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828010.680709:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972887936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828010.680724:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972888064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828010.680768:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828010.680794:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828010/real 1787828010] req@ffff9c1a78861680 x1874668972888064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828010.680821:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972888064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828010.680839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828010.680868:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828010.680871:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972888064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828010.680878:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972888192:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828010.680897:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828010.680907:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828010/real 1787828010] req@ffff9c1a788630c0 x1874668972888192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828010.680950:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972888192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828010.680968:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828010.680974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828010.680981:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972888192:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828010.680995:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972888320:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828010.681046:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828010.681079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828010/real 1787828010] req@ffff9c1a78863c00 x1874668972888320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828010.681120:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972888320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828010.681138:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828010.681153:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828010.681160:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972888320:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828011.191416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828011.191459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828011.447591:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828012.215488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828012.215488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828012.471549:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828013.239477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828013.239477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828013.495539:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828014.263463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828014.263485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828014.519694:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828015.287470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828015.287471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828015.543594:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828015.799497:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828015.799514:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828015.799521:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828015.799538:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4466 00000100:00080000:1.0:1787828015.799554:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828015.799566:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828015.799625:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972888448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828015.799699:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828015.799713:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828015.799716:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828015.799725:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4466 00000100:00080000:1.0:1787828015.799734:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828015.799743:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828015.799760:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972888448:192.168.203.148@tcp:250:kworker.0 00000008:00100000:1.0:1787828015.799761:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828015.799791:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972888576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828015.799808:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828015.799813:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828015.799816:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828015.799821:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4466 00000100:00080000:1.0:1787828015.799826:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828015.799831:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828015.799837:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828015.799847:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828015.799854:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972888704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828015.799870:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828015.799875:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828015.799877:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828015.799883:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4466 00000100:00080000:1.0:1787828015.799889:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828015.799893:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828015.799899:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828015.799904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828015/real 1787828015] req@ffff9c1a78a79e00 x1874668972888448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828015.799917:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972888832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828015.799933:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828015.799939:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972888448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828015.799940:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828015.799946:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828015.799959:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4466 00000100:00080000:1.0:1787828015.799967:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828015.799977:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828015.799983:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828015.799985:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828015.799996:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972888960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828015.800001:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828015.800018:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972888448:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828015.800021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828015.800033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828015.800039:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828015.800047:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4466 00000100:00080000:1.0:1787828015.800052:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828015.800060:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828015.800070:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828015.800076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972888576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828015.800100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972889088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828015.800124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828015.800132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787828015.800134:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828015.800140:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828015.800158:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828015/real 1787828015] req@ffff9c1a78a7b0c0 x1874668972888576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828015.800167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4466 00000100:00080000:1.0:1787828015.800180:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828015.800194:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828015.800195:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972888576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828015.800205:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828015.800216:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828015.800224:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828015.800235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972889216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828015.800235:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972888576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828015.800256:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972888704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828015.800290:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828015.800304:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828015/real 1787828015] req@ffff9c1a78a7bc00 x1874668972888704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828015.800318:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972888704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828015.800328:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828015.800331:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828015.800335:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972888704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828015.800342:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972888832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828015.800399:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828015.800412:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828015/real 1787828015] req@ffff9c1a78a78780 x1874668972888832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828015.800426:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972888832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828015.800437:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828015.800440:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828015.800444:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972888832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828015.800451:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972888960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828015.800469:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828015.800480:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828015/real 1787828015] req@ffff9c1a78a7a1c0 x1874668972888960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828015.800493:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972888960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828015.800502:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828015.800506:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828015.800509:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972888960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828015.800530:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972889088:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828015.800549:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828015.800559:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828015/real 1787828015] req@ffff9c1a78a783c0 x1874668972889088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828015.800572:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972889088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828015.800582:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828015.800586:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828015.800590:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972889088:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828015.800596:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972889216:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828015.800611:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828015.800620:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828015/real 1787828015] req@ffff9c1a78a7a940 x1874668972889216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828015.800641:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972889216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828015.800676:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828015.800681:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828015.800684:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972889216:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828016.311466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828016.311503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828016.567477:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828017.335518:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828017.335518:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828017.591545:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828018.359479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828018.359497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828018.615609:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828019.383434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828019.383434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828019.639465:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828020.407411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828020.407430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828020.663519:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828020.919413:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828020.919425:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828020.919430:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828020.919443:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4471 00000100:00080000:1.0:1787828020.919455:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828020.919465:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828020.919502:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972889344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828020.919529:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828020.919532:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828020.919534:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828020.919537:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4471 00000100:00080000:1.0:1787828020.919540:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828020.919544:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828020.919553:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828020.919563:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972889472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828020.919571:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828020.919575:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828020.919577:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828020.919582:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4471 00000100:00080000:1.0:1787828020.919587:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828020.919593:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828020.919599:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828020.919611:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972889344:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828020.919626:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972889600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828020.919649:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828020.919652:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828020.919654:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828020.919658:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4471 00000100:00080000:1.0:1787828020.919679:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828020.919686:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828020.919694:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828020.919712:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972889728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828020.919724:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828020.919734:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828020.919736:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828020.919742:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828020.919749:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4471 00000100:00080000:1.0:1787828020.919755:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828020.919761:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828020.919767:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828020.919782:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972889856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828020.919795:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828020.919799:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828020.919802:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828020.919809:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4471 00000100:00100000:2.0:1787828020.919813:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828020/real 1787828020] req@ffff9c1a78a78f00 x1874668972889344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828020.919814:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828020.919826:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828020.919839:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828020.919870:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972889344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828020.919874:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972889984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828020.919916:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828020.919921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828020.919928:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828020.919934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828020.919943:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828020.919944:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972889344:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828020.919960:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4471 00000100:00100000:2.0:1787828020.919967:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972889472:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828020.919968:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828020.919983:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828020.919991:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828020.920021:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828020.920026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972890112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828020.920044:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828020/real 1787828020] req@ffff9c1a78a7a580 x1874668972889472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828020.920074:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972889472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828020.920082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828020.920085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828020.920088:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972889472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828020.920110:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972889600:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828020.920124:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828020.920132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828020/real 1787828020] req@ffff9c1a78a78000 x1874668972889600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828020.920140:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972889600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828020.920146:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828020.920149:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828020.920151:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972889600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828020.920155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972889728:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828020.920165:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828020.920172:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828020/real 1787828020] req@ffff9c1a78a7b480 x1874668972889728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828020.920180:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972889728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828020.920187:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828020.920189:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828020.920191:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972889728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828020.920195:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972889856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828020.920206:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828020.920212:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828020/real 1787828020] req@ffff9c1a78a792c0 x1874668972889856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828020.920220:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972889856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828020.920226:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828020.920228:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828020.920230:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972889856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828020.920233:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972889984:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828020.920245:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828020.920251:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828020/real 1787828020] req@ffff9c1a78a7b840 x1874668972889984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828020.920259:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972889984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828020.920265:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828020.920267:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828020.920269:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972889984:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828020.920273:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972890112:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828020.920284:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828020.920290:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828020/real 1787828020] req@ffff9c1a78a79a40 x1874668972890112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828020.920298:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972890112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828020.920304:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828020.920306:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828020.920308:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972890112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828021.431468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828021.431491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828021.687547:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828022.455458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828022.455491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828022.711664:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828023.479476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828023.479516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828023.735641:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828024.503415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828024.503427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828024.759485:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828025.527494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828025.527512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828025.783561:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828026.039474:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828026.039499:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828026.039506:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828026.039520:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4476 00000100:00080000:1.0:1787828026.039535:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828026.039548:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828026.039633:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972890240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828026.039673:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828026.039678:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828026.039681:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828026.039686:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4476 00000100:00080000:1.0:1787828026.039692:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828026.039699:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828026.039711:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828026.039730:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972890240:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828026.039737:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972890368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828026.039764:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828026.039768:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828026.039772:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828026.039776:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4476 00000100:00080000:1.0:1787828026.039781:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828026.039786:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828026.039792:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828026.039807:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972890496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828026.039822:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828026.039827:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828026.039830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828026.039835:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828026.039842:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4476 00000100:00080000:1.0:1787828026.039848:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828026.039854:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828026.039861:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828026.039877:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972890624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828026.039882:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828026/real 1787828026] req@ffff9c1a78a79a40 x1874668972890240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828026.039910:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828026.039915:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828026.039926:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828026.039935:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972890240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828026.039936:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4476 00000100:00080000:1.0:1787828026.039945:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828026.039955:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828026.039964:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828026.039965:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828026.039974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828026.039984:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972890752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828026.039997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972890240:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828026.040003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828026.040013:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828026.040016:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828026.040022:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4476 00000100:00080000:1.0:1787828026.040025:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828026.040029:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828026.040034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828026.040050:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972890368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828026.040051:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972890880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828026.040084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828026.040088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828026.040090:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828026.040093:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4476 00000100:00080000:1.0:1787828026.040102:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828026.040107:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828026.040113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828026.040115:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828026.040134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972891008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828026.040142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828026/real 1787828026] req@ffff9c1a78a7b840 x1874668972890368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828026.040173:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972890368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828026.040190:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828026.040197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828026.040203:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972890368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828026.040216:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972890496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828026.040261:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828026.040280:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828026/real 1787828026] req@ffff9c1a78a792c0 x1874668972890496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828026.040295:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972890496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828026.040305:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828026.040309:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828026.040315:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972890496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828026.040323:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972890624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828026.040419:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828026.040467:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828026/real 1787828026] req@ffff9c1a78a7b480 x1874668972890624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828026.040488:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972890624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828026.040500:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828026.040504:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828026.040510:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972890624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828026.040519:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972890752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828026.040543:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828026.040555:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828026/real 1787828026] req@ffff9c1a78a78000 x1874668972890752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828026.040570:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972890752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828026.040621:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828026.040625:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828026.040628:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972890752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828026.040650:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972890880:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828026.040674:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828026.040685:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828026/real 1787828026] req@ffff9c1a78a7a580 x1874668972890880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828026.040697:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972890880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828026.040706:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828026.040709:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828026.040712:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972890880:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828026.040718:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972891008:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828026.040732:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828026.040741:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828026/real 1787828026] req@ffff9c1a78a78f00 x1874668972891008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828026.040751:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972891008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828026.040760:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828026.040762:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828026.040765:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972891008:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828026.551421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828026.551431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828026.807503:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828027.575417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828027.575417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828027.831603:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828028.599406:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828028.599483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828028.855550:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828029.623465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828029.623499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828029.879598:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828030.647494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828030.647516:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828030.903607:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828031.159598:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828031.159611:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828031.159617:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828031.159630:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4481 00000100:00080000:2.0:1787828031.159654:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828031.159664:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828031.159716:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972891136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828031.159758:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828031.159763:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828031.159777:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828031.159782:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4481 00000100:00080000:2.0:1787828031.159787:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828031.159791:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828031.159804:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828031.159820:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972891264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828031.159836:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828031.159842:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828031.159843:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972891136:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828031.159848:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828031.159860:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4481 00000100:00080000:2.0:1787828031.159865:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828031.159870:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828031.159877:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828031.159892:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972891392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828031.159908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828031.159913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828031.159915:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828031.159921:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4481 00000100:00080000:2.0:1787828031.159926:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828031.159930:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828031.159937:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828031.159938:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828031.159969:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972891520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828031.159990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828031.159999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828031.160001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828031/real 1787828031] req@ffff9c1a49d8d680 x1874668972891136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828031.160007:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828031.160027:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4481 00000100:00080000:2.0:1787828031.160039:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828031.160046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972891136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828031.160057:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828031.160075:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828031.160092:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828031.160102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972891648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828031.160111:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828031.160128:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972891136:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828031.160138:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828031.160147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828031.160152:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828031.160164:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4481 00000100:00080000:2.0:1787828031.160174:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828031.160183:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972891264:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828031.160186:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828031.160208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828031.160239:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972891776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828031.160245:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828031.160273:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828031/real 1787828031] req@ffff9c1a49d8ed00 x1874668972891264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828031.160276:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828031.160296:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828031.160303:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828031.160315:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4481 00000100:00100000:1.0:1787828031.160318:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972891264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828031.160323:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828031.160338:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828031.160342:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828031.160383:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828031.160389:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972891264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828031.160399:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972891392:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787828031.160410:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828031.160435:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828031.160442:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972891904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828031.160459:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828031/real 1787828031] req@ffff9c1a49d8fc00 x1874668972891392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828031.160492:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972891392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828031.160528:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828031.160533:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828031.160537:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972891392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828031.160657:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972891520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828031.160691:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828031.160708:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828031/real 1787828031] req@ffff9c1a49d8e580 x1874668972891520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828031.160723:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972891520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828031.160734:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828031.160738:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828031.160742:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972891520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828031.160749:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972891648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828031.160769:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828031.160780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828031/real 1787828031] req@ffff9c1a49d8c3c0 x1874668972891648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828031.160802:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972891648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828031.160812:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828031.160815:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828031.160820:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972891648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828031.160845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972891776:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828031.160866:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828031.160877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828031/real 1787828031] req@ffff9c1a49d8e1c0 x1874668972891776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828031.160891:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972891776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828031.160901:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828031.160904:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828031.160908:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972891776:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828031.160914:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972891904:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828031.160932:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828031.160943:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828031/real 1787828031] req@ffff9c1a49d8cb40 x1874668972891904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828031.160957:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972891904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828031.160967:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828031.160970:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828031.160973:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972891904:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828031.671502:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828031.671512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828031.927673:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828032.695467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828032.695467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828032.951651:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828033.719420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828033.719433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828033.975559:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828034.743498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828034.743498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828034.999582:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828035.767481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828035.767482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828036.023753:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828036.279444:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828036.279459:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828036.279466:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828036.279479:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4486 00000100:00080000:2.0:1787828036.279494:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828036.279523:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828036.279575:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972892032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828036.279622:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828036.279626:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828036.279628:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828036.279633:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4486 00000100:00080000:2.0:1787828036.279637:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828036.279642:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828036.279652:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828036.279666:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972892160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828036.279678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828036.279682:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828036.279684:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828036.279688:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4486 00000100:00080000:2.0:1787828036.279692:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828036.279698:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972892032:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828036.279700:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828036.279711:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828036.279721:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972892288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828036.279734:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828036.279737:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828036.279740:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828036.279743:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4486 00000100:00080000:2.0:1787828036.279749:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828036.279753:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828036.279759:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828036.279771:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972892416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828036.279781:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828036.279784:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828036.279786:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828036.279790:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4486 00000100:00080000:2.0:1787828036.279795:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828036.279798:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828036.279802:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828036.279813:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972892544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828036.279823:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828036.279827:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828036.279829:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828036.279837:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972892160:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828036.279837:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4486 00000100:00080000:2.0:1787828036.279840:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828036.279852:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828036.279859:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828036.279878:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972892672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828036.279896:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828036.279901:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828036.279906:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828036.279907:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972892288:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828036.279923:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4486 00000100:00080000:2.0:1787828036.279932:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828036.279938:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828036.279945:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828036.279962:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972892416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828036.279977:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972892800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828036.279992:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972892544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828036.280044:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972892672:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828036.280088:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972892800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828036.791430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828036.791459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828037.047599:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828037.815533:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828037.815536:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828038.071638:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828038.839516:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828038.839516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828039.095599:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828039.353698:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787828039.353776:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828039.353814:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828039.353878:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828039.353889:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828039.353900:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828039.353914:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828039.353927:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828039.353944:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787828039.353967:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828036/real 1787828039] req@ffff9c1a49d8c780 x1874668972892288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.354020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972892288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.354056:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.354066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828039.354074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972892288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.354089:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828036/real 1787828039] req@ffff9c1a49d8de00 x1874668972892416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.354117:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972892416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.354134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.354142:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828039.354149:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972892416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.354164:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828036/real 1787828039] req@ffff9c1a49d8c000 x1874668972892544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.354192:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972892544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.354209:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.354215:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828039.354222:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972892544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.354252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828036/real 1787828039] req@ffff9c1a49d8da40 x1874668972892672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.354279:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972892672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.354295:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.354302:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828039.354309:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972892672:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828039.354323:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828036/real 1787828039] req@ffff9c1a49d8cf00 x1874668972892032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.354398:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972892032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.354414:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.354462:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828039.354469:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972892032:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828039.354482:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828036/real 1787828039] req@ffff9c1a49d8f0c0 x1874668972892160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.354538:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972892160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.354554:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.354560:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828039.354567:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972892160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.354637:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828036/real 1787828039] req@ffff9c1a49d8cb40 x1874668972892800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.354670:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972892800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.354687:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.354694:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828039.354700:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972892800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828039.863458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828039.863539:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828040.119670:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828040.887578:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828040.887583:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828041.143695:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828041.399604:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828041.399617:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828041.399622:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828041.399638:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4491 00000100:00080000:1.0:1787828041.399664:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828041.399675:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828041.399724:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972892928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828041.399768:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828041.399776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828041.399779:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828041.399787:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4491 00000100:00080000:1.0:1787828041.399794:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828041.399803:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828041.399821:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828041.399855:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972893056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828041.399857:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972892928:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828041.399903:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828041.399908:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828041.399911:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828041.399917:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4491 00000100:00080000:1.0:1787828041.399924:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828041.399933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828041.399943:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828041.399971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972893184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828041.399993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828041.400027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828041.400032:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828041.400038:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4491 00000100:00080000:1.0:1787828041.400046:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828041.400060:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828041.400061:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828041.400079:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828041.400113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972893312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828041.400136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828041.400143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828041.400147:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828041.400155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828041/real 1787828041] req@ffff9c1a78a7a940 x1874668972892928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828041.400162:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4491 00000100:00080000:1.0:1787828041.400175:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828041.400187:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828041.400200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828041.400216:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972892928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828041.400231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972893440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828041.400247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828041.400259:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828041.400266:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828041.400272:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828041.400281:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828041.400283:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972892928:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828041.400297:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4491 00000100:00080000:1.0:1787828041.400310:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828041.400319:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828041.400330:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828041.400386:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972893056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828041.400447:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828041.400452:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972893568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828041.400508:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828041.400518:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828041/real 1787828041] req@ffff9c1a78a783c0 x1874668972893056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828041.400520:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828041.400527:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828041.400546:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4491 00000100:00080000:1.0:1787828041.400563:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828041.400582:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828041.400584:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972893056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828041.400595:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828041.400620:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828041.400632:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828041.400640:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972893696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828041.400650:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972893056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828041.400680:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972893184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828041.400733:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828041.400758:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828041/real 1787828041] req@ffff9c1a78a7a1c0 x1874668972893184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828041.400797:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972893184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828041.400814:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828041.400822:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828041.400829:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972893184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828041.400842:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972893312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828041.400884:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828041.400909:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828041/real 1787828041] req@ffff9c1a78a78780 x1874668972893312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828041.400935:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972893312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828041.400953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828041.400960:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828041.400966:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972893312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828041.400979:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972893440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828041.401019:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828041.401045:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828041/real 1787828041] req@ffff9c1a78a7bc00 x1874668972893440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828041.401072:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972893440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828041.401103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828041.401111:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828041.401118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972893440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828041.401159:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972893568:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828041.401213:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828041.401237:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828041/real 1787828041] req@ffff9c1a78a7b0c0 x1874668972893568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828041.401274:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972893568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828041.401290:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828041.401296:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828041.401302:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972893568:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828041.401313:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972893696:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828041.401348:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828041.401407:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828041/real 1787828041] req@ffff9c1a78a79e00 x1874668972893696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828041.401431:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972893696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828041.401447:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828041.401452:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828041.401459:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972893696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828041.911436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828041.911463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828042.167538:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828042.935450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828042.935450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828043.191634:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828043.959530:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828043.959564:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828044.215617:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828044.983404:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828044.983415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828045.239570:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828046.007501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828046.007501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828046.263623:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828046.519581:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828046.519616:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828046.519624:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828046.519642:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4496 00000100:00080000:1.0:1787828046.519661:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828046.519674:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828046.522003:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972893824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828046.522065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828046.522073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828046.522079:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828046.522090:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4496 00000100:00080000:1.0:1787828046.522120:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828046.522134:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828046.522154:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828046.522191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972893952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828046.522209:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972893824:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828046.522212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828046.522226:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828046.522232:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828046.522240:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4496 00000100:00080000:1.0:1787828046.522245:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828046.522250:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828046.522257:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828046.522274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972894080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828046.522289:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828046.522293:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828046.522296:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828046.522300:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4496 00000100:00080000:1.0:1787828046.522305:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828046.522310:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828046.522315:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828046.522326:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828046.522333:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972894208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828046.522391:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828046.522398:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828046.522402:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828046.522409:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4496 00000100:00080000:1.0:1787828046.522417:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828046.522425:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828046.522455:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828046.522485:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972894336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828046.522506:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828046.522514:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828046.522523:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828046.522523:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828046/real 1787828046] req@ffff9c1a78a79e00 x1874668972893824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828046.522538:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4496 00000100:00080000:1.0:1787828046.522550:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828046.522559:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828046.522569:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828046.522571:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972893824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.522592:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828046.522594:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972894464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828046.522614:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828046.522631:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972893824:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828046.522632:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828046.522646:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828046.522649:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828046.522668:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4496 00000100:00080000:1.0:1787828046.522675:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828046.522677:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972893952:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828046.522686:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828046.522704:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828046.522724:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972894592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828046.522738:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828046.522770:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828046/real 1787828046] req@ffff9c1a78a7b0c0 x1874668972893952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.522790:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972893952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.522802:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.522808:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828046.522825:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972893952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828046.522833:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972894080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828046.522854:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828046.522866:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828046/real 1787828046] req@ffff9c1a78a7bc00 x1874668972894080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.522880:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972894080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.522890:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.522893:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828046.522897:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972894080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828046.522903:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972894208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828046.522922:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828046.522935:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828046/real 1787828046] req@ffff9c1a78a78780 x1874668972894208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.522949:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972894208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.522959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.522963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828046.522966:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972894208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828046.522973:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972894336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828046.522992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828046.523003:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828046/real 1787828046] req@ffff9c1a78a7a1c0 x1874668972894336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.523018:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972894336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.523027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.523030:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828046.523034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972894336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828046.523041:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972894464:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828046.523070:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828046.523081:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828046/real 1787828046] req@ffff9c1a78a783c0 x1874668972894464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.523095:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972894464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.523106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.523109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828046.523113:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972894464:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828046.523161:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972894592:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828046.523201:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828046.523224:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828046/real 1787828046] req@ffff9c1a78a7a940 x1874668972894592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828046.523287:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972894592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828046.523298:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828046.523302:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828046.523306:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972894592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828047.031479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828047.031484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828047.287581:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828048.055482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828048.055508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828048.311512:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828049.079466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828049.079470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828049.335547:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828050.103458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828050.103460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828050.359540:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828051.127521:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828051.127592:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828051.383672:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828051.639510:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828051.639526:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828051.639534:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828051.639549:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4501 00000100:00080000:2.0:1787828051.639566:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828051.639579:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828051.639637:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972894720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828051.639687:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828051.639694:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828051.639697:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828051.639704:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4501 00000100:00080000:2.0:1787828051.639712:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828051.639720:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828051.639733:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828051.639756:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972894720:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828051.639757:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972894848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828051.639794:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828051.639799:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828051.639802:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828051.639806:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4501 00000100:00080000:2.0:1787828051.639811:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828051.639815:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828051.639821:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828051.639835:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972894976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828051.639848:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828051.639852:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828051.639855:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828051.639859:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4501 00000100:00080000:2.0:1787828051.639864:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828051.639873:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787828051.639874:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787828051.639891:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828051.639909:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972895104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828051.639922:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828051.639926:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828051.639930:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828051.639936:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4501 00000100:00080000:2.0:1787828051.639944:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828051.639945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828051/real 1787828051] req@ffff9c1a49d8e1c0 x1874668972894720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828051.639963:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828051.639980:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828051.639981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972894720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828051.640007:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828051.640013:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828051.640022:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972894720:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828051.640025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972895232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828051.640062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972894848:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828051.640075:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828051.640085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828051.640091:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828051.640092:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828051.640108:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4501 00000100:00100000:1.0:1787828051.640112:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828051/real 1787828051] req@ffff9c1a49d8c3c0 x1874668972894848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828051.640126:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828051.640132:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972894848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828051.640144:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828051.640167:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828051.640171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828051.640176:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828051.640188:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972894848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828051.640199:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972894976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828051.640209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972895360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828051.640233:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828051.640234:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828051.640245:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828051.640250:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828051.640260:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828051/real 1787828051] req@ffff9c1a49d8e580 x1874668972894976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828051.640267:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4501 00000100:00080000:2.0:1787828051.640282:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828051.640291:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972894976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828051.640297:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828051.640329:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828051.640331:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828051.640338:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828051.640348:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972894976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828051.640356:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972895104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828051.640454:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828051.640472:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828051/real 1787828051] req@ffff9c1a49d8fc00 x1874668972895104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828051.640487:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972895488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828051.640498:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972895104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828051.640553:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828051.640560:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828051.640564:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972895104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828051.640590:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972895232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828051.640610:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828051.640618:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828051/real 1787828051] req@ffff9c1a49d8ed00 x1874668972895232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828051.640626:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972895232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828051.640631:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828051.640634:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828051.640636:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972895232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828051.640640:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972895360:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828051.640651:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828051.640657:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828051/real 1787828051] req@ffff9c1a49d8d680 x1874668972895360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828051.640664:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972895360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828051.640669:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828051.640671:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828051.640673:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972895360:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828051.640678:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972895488:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828051.640687:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828051.640693:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828051/real 1787828051] req@ffff9c1a49d8e1c0 x1874668972895488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828051.640701:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972895488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828051.640706:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828051.640708:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828051.640710:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972895488:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828052.151481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828052.151523:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828052.407592:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828053.175474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828053.175505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828053.431582:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828054.199480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828054.199507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828054.455558:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828055.223485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828055.223489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828055.479609:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828056.247492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828056.247493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828056.503581:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828056.759536:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828056.759552:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828056.759560:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828056.759575:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4507 00000100:00080000:0.0:1787828056.759608:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828056.759619:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828056.759682:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972895616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828056.759725:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828056.759731:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828056.759733:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828056.759739:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4507 00000100:00080000:0.0:1787828056.759743:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828056.759749:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828056.759765:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828056.759785:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972895744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828056.759801:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828056.759806:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828056.759809:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828056.759813:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4507 00000100:00080000:0.0:1787828056.759819:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828056.759824:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828056.759837:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828056.759839:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972895616:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828056.759852:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972895872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828056.759868:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828056.759874:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828056.759877:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828056.759884:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4507 00000100:00080000:0.0:1787828056.759888:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828056.759892:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828056.759898:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828056.759915:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972896000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828056.759927:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828056.759933:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828056.759936:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828056.759946:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4507 00000800:00000100:1.0:1787828056.759959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828056.759962:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828056.759970:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828056.759981:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828056.759998:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972896128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828056.760013:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828056.760019:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828056.760030:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828056.760030:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828056/real 1787828056] req@ffff9c1a78860000 x1874668972895616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828056.760056:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4507 00000100:00080000:0.0:1787828056.760069:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828056.760079:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828056.760087:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828056.760089:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972895616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828056.760107:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972896256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828056.760111:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828056.760134:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828056.760149:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828056.760160:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828056.760163:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828056.760170:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972895616:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828056.760172:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4507 00000100:00080000:0.0:1787828056.760189:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828056.760201:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828056.760211:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828056.760231:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972895744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828056.760240:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972896384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828056.760282:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828056.760305:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828056/real 1787828056] req@ffff9c1a78863c00 x1874668972895744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828056.760321:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972895744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828056.760335:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828056.760341:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828056.760413:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972895744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828056.760438:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972895872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828056.760468:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828056.760482:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828056/real 1787828056] req@ffff9c1a788630c0 x1874668972895872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828056.760495:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972895872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828056.760504:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828056.760507:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828056.760511:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972895872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828056.760518:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972896000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828056.760535:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828056.760545:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828056/real 1787828056] req@ffff9c1a78861680 x1874668972896000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828056.760563:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972896000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828056.760577:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828056.760582:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828056.760589:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972896000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828056.760600:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972896128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828056.760633:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828056.760651:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828056/real 1787828056] req@ffff9c1a78862580 x1874668972896128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828056.760663:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972896128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828056.760672:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828056.760676:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828056.760679:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972896128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828056.760686:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972896256:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828056.760708:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828056.760723:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828056/real 1787828056] req@ffff9c1a78861e00 x1874668972896256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828056.760738:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972896256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828056.760749:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828056.760754:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828056.760757:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972896256:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828056.760783:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972896384:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828056.760818:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828056.760838:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828056/real 1787828056] req@ffff9c1a78860780 x1874668972896384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828056.760859:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972896384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828056.760869:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828056.760872:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828056.760876:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972896384:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828057.271467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828057.271548:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828057.527546:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828058.295500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828058.295532:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828058.551624:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828059.319484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828059.319484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828059.575585:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828060.343453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828060.343543:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828060.599643:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828061.367496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828061.367498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828061.623697:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828061.879451:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828061.879463:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828061.879468:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828061.879479:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4512 00000100:00080000:0.0:1787828061.879492:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828061.879502:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828061.879544:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972896512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828061.879572:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828061.879577:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828061.879578:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828061.879583:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4512 00000100:00080000:0.0:1787828061.879587:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828061.879591:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828061.879602:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828061.879620:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972896640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828061.879625:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972896512:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828061.879641:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828061.879645:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828061.879647:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828061.879652:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4512 00000100:00080000:0.0:1787828061.879657:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828061.879662:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828061.879668:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828061.879684:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972896768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828061.879697:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828061.879702:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828061.879704:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828061.879708:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4512 00000100:00080000:0.0:1787828061.879714:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828061.879715:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828061.879724:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828061.879731:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828061.879743:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972896896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828061.879752:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828061.879755:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828061.879757:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828061.879761:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4512 00000100:00080000:0.0:1787828061.879765:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828061.879767:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828061/real 1787828061] req@ffff9c1a788621c0 x1874668972896512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828061.879771:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828061.879776:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828061.879787:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972897024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828061.879816:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828061.879819:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828061.879821:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828061.879827:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972896512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828061.879831:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4512 00000100:00080000:0.0:1787828061.879839:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828061.879848:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828061.879855:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828061.879856:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828061.879867:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828061.879873:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972897152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828061.879878:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972896512:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828061.879897:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828061.879901:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828061.879904:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828061.879919:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972896640:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828061.879922:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4512 00000100:00080000:0.0:1787828061.879932:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828061.879941:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828061.879950:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828061.879976:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828061.880004:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972897280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828061.880005:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828061/real 1787828061] req@ffff9c1a78862940 x1874668972896640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828061.880080:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972896640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828061.880092:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828061.880098:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828061.880105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972896640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828061.880116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972896768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828061.880156:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828061.880180:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828061/real 1787828061] req@ffff9c1a78861a40 x1874668972896768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828061.880197:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972896768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828061.880209:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828061.880213:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828061.880219:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972896768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828061.880226:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972896896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828061.880248:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828061.880261:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828061/real 1787828061] req@ffff9c1a78860f00 x1874668972896896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828061.880274:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972896896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828061.880293:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828061.880297:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828061.880300:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972896896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828061.880307:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972897024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828061.880325:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828061.880416:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828061/real 1787828061] req@ffff9c1a78862d00 x1874668972897024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828061.880449:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972897024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828061.880462:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828061.880466:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828061.880471:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972897024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828061.880483:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972897152:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828061.880518:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828061.880537:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828061/real 1787828061] req@ffff9c1a78860b40 x1874668972897152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828061.880555:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972897152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828061.880581:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828061.880584:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828061.880588:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972897152:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828061.880607:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972897280:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828061.880624:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828061.880633:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828061/real 1787828061] req@ffff9c1a78863840 x1874668972897280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828061.880644:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972897280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828061.880652:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828061.880655:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828061.880658:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972897280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828062.391458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828062.391465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828062.647618:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828063.415440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828063.415472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828063.671534:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828064.439471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828064.439539:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828064.695597:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828065.463500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828065.463516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828065.719575:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828066.487456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828066.487460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828066.743558:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828066.999518:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828066.999533:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828066.999538:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828066.999549:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4517 00000100:00080000:1.0:1787828066.999561:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828066.999569:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828066.999611:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972897408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828066.999646:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828066.999649:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828066.999651:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828066.999655:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4517 00000100:00080000:1.0:1787828066.999660:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828066.999664:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828066.999674:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828066.999686:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972897536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828066.999696:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828066.999700:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828066.999702:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828066.999705:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4517 00000100:00080000:1.0:1787828066.999708:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828066.999711:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828066.999715:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828066.999725:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972897664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828066.999738:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828066.999742:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828066.999747:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828066.999759:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972897408:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828066.999761:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4517 00000100:00080000:1.0:1787828066.999768:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828066.999774:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828066.999778:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828066.999790:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972897792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828066.999816:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828066.999820:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828066.999822:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828066.999825:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4517 00000100:00080000:1.0:1787828066.999830:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828066.999833:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828066.999838:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828066.999845:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828066.999857:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972897920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828066.999869:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828066.999872:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828066.999874:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828066.999878:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4517 00000100:00080000:1.0:1787828066.999882:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828066.999887:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828066.999892:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828066.999897:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828066/real 1787828066] req@ffff9c1a78a78f00 x1874668972897408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828066.999909:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972898048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828066.999922:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828066.999928:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972897408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828066.999929:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828066.999934:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828066.999944:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4517 00000100:00080000:1.0:1787828066.999952:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828066.999956:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828066.999958:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828066.999974:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828066.999978:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828066.999986:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972897408:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828066.999992:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972898176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828067.000001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972897536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828067.000036:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828067.000050:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828066/real 1787828067] req@ffff9c1a78a7a580 x1874668972897536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.000062:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972897536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.000070:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.000073:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828067.000076:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972897536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828067.000101:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972897664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828067.000116:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828067.000125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828066/real 1787828067] req@ffff9c1a78a78000 x1874668972897664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.000136:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972897664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.000144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.000146:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828067.000150:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972897664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828067.000155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972897792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828067.000170:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828067.000179:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828066/real 1787828067] req@ffff9c1a78a7b480 x1874668972897792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.000200:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972897792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.000208:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.000211:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828067.000213:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972897792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828067.000218:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972897920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828067.000232:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828067.000240:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828066/real 1787828067] req@ffff9c1a78a792c0 x1874668972897920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.000251:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972897920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.000258:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.000261:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828067.000263:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972897920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828067.000268:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972898048:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828067.000319:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828067.000331:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828066/real 1787828067] req@ffff9c1a78a7b840 x1874668972898048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.000386:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972898048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.000395:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.000398:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828067.000401:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972898048:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828067.000407:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972898176:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828067.000427:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828067.000437:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828067/real 1787828067] req@ffff9c1a78a79a40 x1874668972898176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828067.000448:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972898176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828067.000457:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828067.000460:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828067.000463:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972898176:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828067.511480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828067.511538:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828067.767493:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828068.535464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828068.535476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828068.791591:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828069.559463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828069.559472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828069.815606:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828070.583457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828070.583500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828070.839506:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828071.607418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828071.607428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828071.863578:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828072.119470:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828072.119484:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828072.119491:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828072.119505:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4522 00000100:00080000:1.0:1787828072.119518:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828072.119529:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828072.119579:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972898304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828072.119617:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828072.119623:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828072.119627:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828072.119634:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4522 00000100:00080000:1.0:1787828072.119642:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828072.119650:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828072.119665:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828072.119668:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972898304:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828072.119692:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972898432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828072.119720:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828072.119726:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828072.119730:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828072.119735:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4522 00000100:00080000:1.0:1787828072.119743:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828072.119749:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828072.119774:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828072.119775:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828072.119811:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972898560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828072.119827:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828072.119832:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828072.119835:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828072.119840:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4522 00000100:00080000:1.0:1787828072.119844:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828072.119849:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828072.119856:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828072.119865:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff9c1a78a79a40 x1874668972898304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828072.119874:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972898688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828072.119905:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972898304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828072.119925:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828072.119950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828072.119953:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828072.119953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.119968:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828072.119969:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4522 00000100:00080000:1.0:1787828072.119979:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828072.119981:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972898304:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828072.119988:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828072.119996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828072.120009:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972898816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828072.120019:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972898432:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828072.120023:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828072.120029:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828072.120031:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828072.120036:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4522 00000100:00080000:1.0:1787828072.120042:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828072.120046:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828072.120052:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828072.120060:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828072.120067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972898944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828072.120078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828072.120082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828072.120084:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828072.120089:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff9c1a78a7b840 x1874668972898432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828072.120091:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4522 00000100:00080000:1.0:1787828072.120098:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828072.120106:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828072.120116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828072.120132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972899072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828072.120142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972898432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.120182:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.120187:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828072.120191:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972898432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828072.120200:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972898560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828072.120251:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828072.120311:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff9c1a78a792c0 x1874668972898560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828072.120325:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972898560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.120340:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.120344:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828072.120347:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972898560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828072.120353:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972898688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828072.120405:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828072.120417:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff9c1a78a7b480 x1874668972898688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828072.120428:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972898688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.120437:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.120440:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828072.120444:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972898688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828072.120463:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972898816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828072.120480:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828072.120489:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff9c1a78a78000 x1874668972898816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828072.120500:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972898816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.120507:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.120510:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828072.120514:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972898816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828072.120519:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972898944:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828072.120533:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828072.120542:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff9c1a78a7a580 x1874668972898944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828072.120552:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972898944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.120559:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.120562:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828072.120565:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972898944:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828072.120570:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972899072:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828072.120583:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828072.120591:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828072/real 1787828072] req@ffff9c1a78a78f00 x1874668972899072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828072.120602:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972899072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828072.120609:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828072.120612:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828072.120614:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972899072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828072.631460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828072.631465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828072.887479:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828073.655458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828073.655459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828073.911545:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828074.679467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828074.679499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828074.935463:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828075.703500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828075.703522:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828075.959590:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828076.727452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828076.727456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828076.983566:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828077.239532:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828077.239545:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828077.239551:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828077.239575:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4527 00000100:00080000:0.0:1787828077.239588:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828077.239597:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828077.239649:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972899200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828077.239688:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828077.239693:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828077.239695:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828077.239701:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4527 00000100:00080000:0.0:1787828077.239705:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828077.239709:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828077.239721:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828077.239738:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972899328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828077.239753:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828077.239758:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828077.239761:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828077.239765:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4527 00000100:00080000:0.0:1787828077.239770:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828077.239774:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828077.239781:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828077.239789:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972899200:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828077.239801:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972899456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828077.239818:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828077.239832:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828077.239835:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828077.239841:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4527 00000100:00080000:0.0:1787828077.239845:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828077.239849:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828077.239855:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828077.239870:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972899584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828077.239882:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828077.239886:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828077.239888:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828077.239892:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4527 00000100:00080000:0.0:1787828077.239897:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828077.239904:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828077.239910:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828077.239919:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828077.239935:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972899712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828077.239946:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828077.239950:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828077.239952:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828077.239959:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4527 00000100:00080000:0.0:1787828077.239964:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828077.239973:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff9c1a788612c0 x1874668972899200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828077.239974:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828077.239986:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828077.240010:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972899840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828077.240020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972899200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828077.240037:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828077.240052:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828077.240063:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828077.240069:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828077.240071:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4527 00000100:00080000:0.0:1787828077.240082:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828077.240085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828077.240094:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828077.240102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972899200:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787828077.240103:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828077.240132:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972899968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828077.240133:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972899328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828077.240182:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828077.240202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff9c1a78863840 x1874668972899328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828077.240218:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972899328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828077.240265:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828077.240270:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828077.240274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972899328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828077.240308:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972899456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828077.240337:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828077.240393:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff9c1a78860b40 x1874668972899456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828077.240409:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972899456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828077.240420:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828077.240424:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828077.240428:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972899456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828077.240436:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972899584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828077.240460:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828077.240472:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff9c1a78862d00 x1874668972899584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828077.240486:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972899584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828077.240497:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828077.240500:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828077.240504:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972899584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828077.240510:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972899712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828077.240529:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828077.240539:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff9c1a78860f00 x1874668972899712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828077.240563:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972899712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828077.240580:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828077.240583:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828077.240587:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972899712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828077.240593:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972899840:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828077.240612:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828077.240622:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff9c1a78861a40 x1874668972899840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828077.240636:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972899840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828077.240646:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828077.240649:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828077.240653:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972899840:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828077.240659:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972899968:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828077.240678:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828077.240688:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828077/real 1787828077] req@ffff9c1a78862940 x1874668972899968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828077.240702:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972899968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828077.240720:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828077.240723:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828077.240727:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972899968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828077.751467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828077.751493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828078.007533:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828078.775471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828078.775492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828079.031644:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828079.799499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828079.799515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828080.055523:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828080.823452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828080.823558:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828081.079643:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828081.847422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828081.847444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828082.103476:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828082.359490:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828082.359503:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828082.359510:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828082.359524:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4532 00000100:00080000:0.0:1787828082.359552:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828082.359563:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828082.359613:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972900096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828082.359674:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828082.359689:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828082.359692:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828082.359698:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4532 00000100:00080000:0.0:1787828082.359703:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828082.359710:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828082.359723:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828082.359739:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972900224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828082.359754:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828082.359760:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828082.359763:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828082.359768:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4532 00000100:00080000:0.0:1787828082.359774:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828082.359781:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972900096:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828082.359785:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828082.359797:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828082.359816:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972900352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828082.359866:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828082.359873:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828082.359876:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828082.359882:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4532 00000100:00080000:0.0:1787828082.359889:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828082.359901:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828082.359903:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787828082.359915:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828082.359937:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972900480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828082.359954:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828082.359955:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff9c1a788621c0 x1874668972900096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828082.359982:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828082.359988:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828082.359998:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4532 00000100:00080000:0.0:1787828082.360004:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828082.360020:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828082.360024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972900096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828082.360028:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828082.360043:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828082.360050:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828082.360054:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972900608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828082.360058:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972900096:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828082.360073:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972900224:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828082.360081:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828082.360088:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828082.360091:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828082.360099:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4532 00000800:00000100:2.0:1787828082.360105:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828082.360108:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828082.360118:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828082.360122:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff9c1a78860780 x1874668972900224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828082.360131:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828082.360150:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972900224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828082.360156:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972900736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828082.360163:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828082.360173:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828082.360180:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972900224:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828082.360203:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828082.360220:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828082.360223:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828082.360231:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4532 00000100:00080000:0.0:1787828082.360239:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828082.360244:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972900352:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828082.360251:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828082.360262:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828082.360275:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828082.360288:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff9c1a78861e00 x1874668972900352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828082.360295:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972900864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828082.360307:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972900352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828082.360343:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828082.360348:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828082.360353:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972900352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828082.360404:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972900480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828082.360455:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828082.360489:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff9c1a78862580 x1874668972900480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828082.360502:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972900480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828082.360517:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828082.360520:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828082.360523:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972900480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828082.360528:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972900608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828082.360567:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828082.360577:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff9c1a78861680 x1874668972900608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828082.360587:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972900608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828082.360595:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828082.360598:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828082.360601:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972900608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828082.360606:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972900736:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828082.360621:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828082.360629:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff9c1a788630c0 x1874668972900736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828082.360656:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972900736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828082.360663:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828082.360665:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828082.360668:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972900736:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828082.360684:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972900864:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828082.360709:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828082.360716:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828082/real 1787828082] req@ffff9c1a78863c00 x1874668972900864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828082.360726:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972900864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828082.360733:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828082.360735:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828082.360738:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972900864:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828082.871492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828082.871523:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828083.127643:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828083.895449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828083.895459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828084.151609:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828084.919419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828084.919420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828085.175435:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828085.943492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828085.943573:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828086.199574:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828086.967508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828086.967508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828087.223546:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828087.479469:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828087.479482:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828087.479488:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828087.479501:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4537 00000100:00080000:0.0:1787828087.479517:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828087.479529:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828087.479594:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972900992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828087.479647:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828087.479655:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828087.479659:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828087.479668:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4537 00000100:00080000:0.0:1787828087.479676:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828087.479685:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828087.479704:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828087.479732:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972901120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828087.479744:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972900992:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828087.479754:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828087.479763:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828087.479766:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828087.479772:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4537 00000100:00080000:0.0:1787828087.479777:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828087.479781:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828087.479789:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828087.479805:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972901248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828087.479820:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828087.479825:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828087.479827:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828087.479831:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4537 00000100:00080000:0.0:1787828087.479835:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828087.479840:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828087.479846:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828087.479865:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972901376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828087.479867:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828087.479882:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828087.479887:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828087.479889:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828087.479895:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4537 00000100:00080000:0.0:1787828087.479899:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828087.479905:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828087.479912:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828087.479926:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972901504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828087.479938:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828087.479944:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828087.479949:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828087.479959:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828087/real 1787828087] req@ffff9c1a78860000 x1874668972900992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828087.479964:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4537 00000100:00080000:0.0:1787828087.479978:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828087.479993:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828087.480003:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972900992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828087.480007:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828087.480032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828087.480044:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828087.480045:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972901632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828087.480054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972900992:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828087.480080:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828087.480085:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828087.480088:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828087.480096:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4537 00000100:00080000:0.0:1787828087.480103:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828087.480108:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972901120:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828087.480115:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828087.480129:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828087.480148:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972901760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828087.480227:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828087.480263:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828087/real 1787828087] req@ffff9c1a78863c00 x1874668972901120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828087.480290:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972901120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828087.480306:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828087.480312:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828087.480319:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972901120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828087.480330:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972901248:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828087.480445:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828087.480471:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828087/real 1787828087] req@ffff9c1a788630c0 x1874668972901248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828087.480497:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972901248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828087.480513:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828087.480518:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828087.480525:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972901248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828087.480539:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972901376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828087.480585:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828087.480609:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828087/real 1787828087] req@ffff9c1a78861680 x1874668972901376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828087.480635:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972901376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828087.480653:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828087.480660:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828087.480666:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972901376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828087.480677:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972901504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828087.480709:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828087.480725:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828087/real 1787828087] req@ffff9c1a78862580 x1874668972901504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828087.480741:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972901504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828087.480758:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828087.480765:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828087.480771:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972901504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828087.480783:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972901632:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828087.480823:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828087.480846:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828087/real 1787828087] req@ffff9c1a78861e00 x1874668972901632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828087.480886:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972901632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828087.480904:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828087.480911:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828087.480917:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972901632:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828087.480961:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972901760:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828087.481005:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828087.481029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828087/real 1787828087] req@ffff9c1a78860780 x1874668972901760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828087.481046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972901760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828087.481057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828087.481061:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828087.481065:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972901760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828087.991479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828087.991498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828088.247571:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828089.015485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828089.015518:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828089.271633:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828090.039463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828090.039488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828090.295552:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828091.063450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828091.063451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828091.319542:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828092.087460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828092.087530:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828092.343546:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828092.599513:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828092.599526:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828092.599531:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828092.599544:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4542 00000100:00080000:1.0:1787828092.599558:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828092.599568:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828092.599617:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972901888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828092.599657:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828092.599662:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828092.599665:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828092.599670:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4542 00000100:00080000:1.0:1787828092.599674:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828092.599678:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828092.599689:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828092.599704:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972902016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828092.599717:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828092.599721:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828092.599724:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828092.599729:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4542 00000100:00080000:1.0:1787828092.599734:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828092.599739:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828092.599746:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828092.599757:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972902144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828092.599772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828092.599779:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828092.599782:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972901888:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828092.599784:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828092.599802:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4542 00000100:00080000:1.0:1787828092.599807:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828092.599812:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828092.599819:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828092.599835:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972902272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828092.599847:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828092.599851:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828092.599854:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828092.599858:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4542 00000100:00080000:1.0:1787828092.599863:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828092.599868:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828092.599873:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828092.599884:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828092.599892:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972902400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828092.599908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828092.599912:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828092.599915:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828092.599920:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4542 00000100:00080000:1.0:1787828092.599924:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828092.599928:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828092.599933:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828092.599948:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972902528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828092.599959:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828092.599967:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828092.599970:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828092/real 1787828092] req@ffff9c1a78a7a940 x1874668972901888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828092.599971:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828092.599988:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4542 00000100:00080000:1.0:1787828092.599998:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828092.600009:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828092.600017:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828092.600028:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972901888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828092.600038:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972902656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828092.600056:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828092.600066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828092.600074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972901888:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828092.600087:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972902016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828092.600158:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828092.600185:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828092/real 1787828092] req@ffff9c1a78a783c0 x1874668972902016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828092.600212:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972902016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828092.600229:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828092.600237:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828092.600242:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972902016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828092.600277:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972902144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828092.600308:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828092.600323:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828092/real 1787828092] req@ffff9c1a78a7a1c0 x1874668972902144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828092.600338:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972902144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828092.600400:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828092.600404:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828092.600408:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972902144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828092.600417:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972902272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828092.600442:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828092.600456:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828092/real 1787828092] req@ffff9c1a78a78780 x1874668972902272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828092.600471:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972902272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828092.600481:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828092.600485:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828092.600489:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972902272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828092.600496:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972902400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828092.600515:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828092.600526:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828092/real 1787828092] req@ffff9c1a78a7bc00 x1874668972902400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828092.600539:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972902400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828092.600549:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828092.600552:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828092.600557:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972902400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828092.600563:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972902528:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828092.600582:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828092.600593:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828092/real 1787828092] req@ffff9c1a78a7b0c0 x1874668972902528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828092.600606:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972902528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828092.600615:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828092.600619:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828092.600622:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972902528:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828092.600629:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972902656:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828092.600647:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828092.600658:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828092/real 1787828092] req@ffff9c1a78a79e00 x1874668972902656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828092.600672:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972902656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828092.600681:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828092.600684:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828092.600688:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972902656:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828093.111484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828093.111484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828093.367540:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828094.135460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828094.135474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828094.391669:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828095.159498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828095.159541:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828095.415573:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828096.183499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828096.183510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828096.439612:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828097.207448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828097.207484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828097.463575:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828097.719539:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828097.719556:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828097.719565:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828097.719583:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4548 00000100:00080000:2.0:1787828097.719601:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828097.719612:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828097.719672:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972902784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828097.719725:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828097.719733:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828097.719738:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828097.719749:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4548 00000100:00080000:2.0:1787828097.719756:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828097.719766:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828097.719786:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828097.719839:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972902912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828097.719847:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972902784:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828097.719875:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828097.719881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828097.719885:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828097.719893:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4548 00000100:00080000:2.0:1787828097.719901:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828097.719910:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828097.719918:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828097.719941:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972903040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828097.719965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828097.719970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828097.719976:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828097.719981:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4548 00000100:00080000:2.0:1787828097.719986:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828097.719995:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787828097.720000:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787828097.720005:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828097.720025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972903168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828097.720038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828097.720043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828097.720046:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828097.720051:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4548 00000100:00080000:2.0:1787828097.720056:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828097.720061:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828097.720069:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828097.720114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972903296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828097.720128:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828097.720135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828097.720140:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828097.720148:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4548 00000100:00080000:2.0:1787828097.720157:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828097.720167:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828097.720179:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828097.720208:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972903424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828097.720227:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828097.720233:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828097.720236:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828097.720241:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4548 00000100:00080000:2.0:1787828097.720247:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828097.720252:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828097.720259:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828097.720273:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972903552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828097.720678:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828097/real 1787828097] req@ffff9c1a49d8cb40 x1874668972902784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828097.720808:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972902784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828097.720834:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828097.720842:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828097.720852:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972902784:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828097.720892:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972902912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828097.720935:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828097.720956:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828097/real 1787828097] req@ffff9c1a49d8f0c0 x1874668972902912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828097.720973:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972902912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828097.720984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828097.720989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828097.720993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972902912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828097.721001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972903040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828097.721023:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828097.721037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828097/real 1787828097] req@ffff9c1a49d8cf00 x1874668972903040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828097.721051:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972903040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828097.721061:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828097.721064:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828097.721068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972903040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828097.721075:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972903168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828097.721147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828097.721172:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828097/real 1787828097] req@ffff9c1a49d8da40 x1874668972903168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828097.721194:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972903168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828097.721206:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828097.721210:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828097.721214:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972903168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828097.721223:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972903296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828097.721264:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828097.721286:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828097/real 1787828097] req@ffff9c1a49d8c000 x1874668972903296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828097.721308:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972903296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828097.721326:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828097.721332:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828097.721339:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972903296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828097.721390:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972903424:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828097.721430:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828097.721446:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828097/real 1787828097] req@ffff9c1a49d8de00 x1874668972903424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828097.721461:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972903424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828097.721472:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828097.721476:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828097.721480:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972903424:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828097.721487:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972903552:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828097.721507:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828097.721518:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828097/real 1787828097] req@ffff9c1a49d8c780 x1874668972903552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828097.721532:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972903552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828097.721541:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828097.721545:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828097.721550:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972903552:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828098.231501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828098.231505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828098.487613:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828099.255496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828099.255522:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828099.511516:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828100.279493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828100.279565:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828100.535597:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828101.303509:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828101.303550:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828101.559596:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828102.327477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828102.327538:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828102.583667:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828102.839492:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828102.839505:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828102.839511:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828102.839524:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4553 00000100:00080000:3.0:1787828102.839538:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828102.839556:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828102.839608:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972903680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828102.839649:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828102.839654:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828102.839657:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828102.839662:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4553 00000100:00080000:3.0:1787828102.839668:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828102.839673:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828102.839685:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828102.839705:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972903808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828102.839736:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828102.839737:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972903680:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828102.839753:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828102.839771:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828102.839780:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4553 00000100:00080000:3.0:1787828102.839785:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828102.839791:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828102.839798:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828102.839815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972903936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828102.839829:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828102.839834:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828102.839836:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828102.839841:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4553 00000100:00080000:3.0:1787828102.839876:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828102.839881:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828102.839889:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828102.839902:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972904064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828102.839915:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828102.839920:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828102.839923:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828102.839929:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4553 00000100:00080000:3.0:1787828102.839938:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828102.839946:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828102.839954:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828102.839974:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972904192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828102.839985:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972903808:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828102.839994:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828102.840000:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828102.840003:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828102.840011:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4553 00000100:00080000:3.0:1787828102.840016:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828102.840024:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828102.840037:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828102.840077:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972904320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828102.840090:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972903936:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828102.840103:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828102.840112:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828102.840117:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828102.840123:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4553 00000100:00080000:3.0:1787828102.840140:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828102.840146:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828102.840156:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828102.840168:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972904064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828102.840182:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972904448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828102.840223:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972904192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828102.840261:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972904320:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828102.840290:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972904448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828103.351467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828103.351537:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828103.607659:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828104.375475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828104.375604:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828104.631623:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828105.399540:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828105.399550:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828105.655671:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828105.911745:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787828105.911817:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828105.911840:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828105.911895:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828105.911907:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828105.911921:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828105.911949:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828105.911969:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828105.911986:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787828105.912021:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828102/real 1787828105] req@ffff9c1a7895e580 x1874668972904192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828105.912096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972904192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828105.912130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828105.912139:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828105.912147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972904192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828105.912161:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828102/real 1787828105] req@ffff9c1a7895fc00 x1874668972904320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828105.912183:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972904320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828105.912198:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828105.912204:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828105.912211:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972904320:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828105.912222:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828102/real 1787828105] req@ffff9c1a7895e940 x1874668972904448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828105.912258:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972904448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828105.912274:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828105.912281:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828105.912288:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972904448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828105.912301:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828102/real 1787828105] req@ffff9c1a7895f480 x1874668972903808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828105.912324:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972903808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828105.912341:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828105.912392:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828105.912401:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972903808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828105.912415:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828102/real 1787828105] req@ffff9c1a7895cf00 x1874668972903936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828105.912440:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972903936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828105.912456:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828105.912462:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828105.912468:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972903936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828105.912479:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828102/real 1787828105] req@ffff9c1a7895c3c0 x1874668972904064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828105.912500:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972904064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828105.912515:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828105.912521:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828105.912527:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972904064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828105.912542:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828102/real 1787828105] req@ffff9c1a7895c000 x1874668972903680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828105.912581:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972903680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828105.912597:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828105.912604:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828105.912611:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972903680:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828106.423490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828106.423495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828106.679512:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828107.447476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828107.447491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828107.703618:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828107.959457:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828107.959470:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828107.959475:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828107.959489:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4558 00000100:00080000:3.0:1787828107.959504:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828107.959515:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828107.959565:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972904576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828107.959603:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828107.959608:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828107.959611:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828107.959616:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4558 00000100:00080000:3.0:1787828107.959620:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828107.959625:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828107.959637:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828107.959654:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972904704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828107.959675:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828107.959687:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828107.959688:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972904576:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828107.959695:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828107.959711:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4558 00000100:00080000:3.0:1787828107.959719:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828107.959728:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828107.959739:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828107.959764:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972904832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828107.959786:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828107.959793:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828107.959797:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828107.959806:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4558 00000100:00080000:3.0:1787828107.959815:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828107.959827:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828107.959832:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828107.959841:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828107.959869:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972904960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828107.959890:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828107.959897:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828107.959901:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828107.959910:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4558 00000100:00100000:2.0:1787828107.959920:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828107/real 1787828107] req@ffff9c1a7895f840 x1874668972904576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828107.959923:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828107.959940:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828107.959957:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828107.959989:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972905088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828107.959998:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972904576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828107.960058:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828107.960077:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828107.960085:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828107.960098:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4558 00000100:00080000:2.0:1787828107.960100:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828107.960110:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828107.960116:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828107.960129:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828107.960133:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972904576:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787828107.960145:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828107.960176:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972905216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828107.960196:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972904704:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828107.960199:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828107.960214:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828107.960217:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828107.960238:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4558 00000100:00080000:3.0:1787828107.960260:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828107.960263:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828107.960274:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828107.960288:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828107.960303:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828107/real 1787828107] req@ffff9c1a7895d2c0 x1874668972904704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828107.960319:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972905344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828107.960403:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972904704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828107.960431:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828107.960440:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828107.960447:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972904704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828107.960463:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972904832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828107.960510:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828107.960533:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828107/real 1787828107] req@ffff9c1a7895de00 x1874668972904832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828107.960557:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972904832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828107.960574:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828107.960581:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828107.960588:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972904832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828107.960600:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972904960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828107.960637:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828107.960660:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828107/real 1787828107] req@ffff9c1a7895c780 x1874668972904960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828107.960685:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972904960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828107.960701:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828107.960707:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828107.960714:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972904960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828107.960726:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972905088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828107.960764:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828107.960788:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828107/real 1787828107] req@ffff9c1a7895cb40 x1874668972905088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828107.960814:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972905088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828107.960833:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828107.960840:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828107.960846:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972905088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828107.960887:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972905216:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828107.960930:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828107.960955:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828107/real 1787828107] req@ffff9c1a7895f0c0 x1874668972905216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828107.960983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972905216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828107.961001:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828107.961007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828107.961037:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972905216:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828107.961050:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972905344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828107.961091:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828107.961116:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828107/real 1787828107] req@ffff9c1a7895da40 x1874668972905344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828107.961142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972905344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828107.961161:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828107.961168:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828107.961176:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972905344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828108.471427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828108.471448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828108.727590:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828109.495492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828109.495537:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828109.751562:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828110.519507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828110.519514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828110.775570:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828111.543460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828111.543460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828111.799603:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828112.567448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828112.567457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828112.823611:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828113.079525:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828113.079537:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828113.079542:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828113.079556:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4563 00000100:00080000:0.0:1787828113.079570:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828113.079580:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828113.079632:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972905472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828113.079672:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828113.079677:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828113.079680:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828113.079685:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4563 00000100:00080000:0.0:1787828113.079689:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828113.079694:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828113.079708:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828113.079725:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972905600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828113.079740:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828113.079745:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828113.079747:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828113.079756:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972905472:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828113.079758:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4563 00000100:00080000:0.0:1787828113.079768:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828113.079776:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828113.079782:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828113.079800:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972905728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828113.079814:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828113.079819:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828113.079822:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828113.079827:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4563 00000100:00080000:0.0:1787828113.079832:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828113.079836:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828113.079842:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828113.079863:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972905856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828113.079864:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828113.079883:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828113.079887:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828113.079891:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828113.079898:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4563 00000100:00080000:0.0:1787828113.079903:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828113.079907:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828113.079913:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828113.079930:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff9c1a788621c0 x1874668972905472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828113.079932:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972905984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828113.079964:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828113.079968:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828113.079971:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828113.080001:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4563 00000100:00080000:0.0:1787828113.080013:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828113.080019:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828113.080027:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828113.080038:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972905472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828113.080046:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972906112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828113.080067:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828113.080085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828113.080092:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828113.080099:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972905472:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828113.080102:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828113.080109:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828113.080119:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4563 00000100:00080000:0.0:1787828113.080124:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828113.080129:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828113.080135:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828113.080147:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972905600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828113.080156:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972906240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828113.080200:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828113.080223:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff9c1a78862940 x1874668972905600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828113.080240:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972905600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828113.080251:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828113.080255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828113.080259:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972905600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828113.080266:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972905728:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828113.080286:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828113.080297:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff9c1a78861a40 x1874668972905728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828113.080311:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972905728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828113.080330:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828113.080334:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828113.080338:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972905728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828113.080383:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972905856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828113.080408:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828113.080423:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff9c1a78860f00 x1874668972905856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828113.080441:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972905856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828113.080460:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828113.080466:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828113.080473:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972905856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828113.080486:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972905984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828113.080525:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828113.080549:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff9c1a78862d00 x1874668972905984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828113.080568:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972905984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828113.080579:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828113.080584:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828113.080588:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972905984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828113.080596:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972906112:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828113.080624:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828113.080637:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff9c1a78860b40 x1874668972906112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828113.080651:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972906112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828113.080661:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828113.080665:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828113.080669:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972906112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828113.080694:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972906240:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828113.080714:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828113.080725:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828113/real 1787828113] req@ffff9c1a78863840 x1874668972906240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828113.080739:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972906240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828113.080749:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828113.080753:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828113.080756:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972906240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828113.591443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828113.591444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828113.847543:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828114.615494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828114.615494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828114.871536:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828115.639425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828115.639454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828115.895510:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828116.663459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828116.663459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828116.919512:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828117.687456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828117.687521:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828117.943545:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828118.199494:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828118.199506:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828118.199523:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828118.199537:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4568 00000100:00080000:3.0:1787828118.199550:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828118.199561:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828118.199609:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972906368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828118.199645:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828118.199650:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828118.199652:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828118.199658:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4568 00000100:00080000:3.0:1787828118.199662:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828118.199667:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828118.199679:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828118.199697:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972906496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828118.199724:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972906368:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828118.199726:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828118.199744:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828118.199752:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828118.199761:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4568 00000100:00080000:3.0:1787828118.199770:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828118.199779:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828118.199790:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828118.199816:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972906624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828118.199835:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828118.199840:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828118.199847:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828118.199851:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828118.199858:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4568 00000100:00080000:3.0:1787828118.199868:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828118.199874:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828118.199882:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828118.199899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff9c1a7895e1c0 x1874668972906368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828118.199905:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972906752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828118.199980:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828118.199985:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828118.199989:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828118.199992:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972906368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828118.199999:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4568 00000100:00080000:3.0:1787828118.200010:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828118.200021:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828118.200022:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828118.200029:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828118.200034:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828118.200044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972906368:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828118.200053:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972906880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828118.200067:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828118.200071:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828118.200074:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828118.200079:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4568 00000100:00080000:3.0:1787828118.200085:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828118.200086:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972906496:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828118.200096:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828118.200105:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828118.200122:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972907008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828118.200137:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828118.200141:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828118.200145:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828118.200150:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828118.200159:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4568 00000100:00080000:3.0:1787828118.200165:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828118.200172:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff9c1a7895da40 x1874668972906496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828118.200175:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828118.200188:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828118.200206:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972906496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828118.200221:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972907136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828118.200225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828118.200241:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828118.200254:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972906496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828118.200276:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972906624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828118.200325:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828118.200339:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff9c1a7895f0c0 x1874668972906624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828118.200394:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972906624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828118.200405:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828118.200409:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828118.200414:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972906624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828118.200422:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972906752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828118.200447:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828118.200461:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff9c1a7895cb40 x1874668972906752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828118.200475:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972906752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828118.200486:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828118.200490:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828118.200493:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972906752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828118.200500:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972906880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828118.200518:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828118.200529:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff9c1a7895c780 x1874668972906880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828118.200553:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972906880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828118.200563:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828118.200566:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828118.200570:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972906880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828118.200592:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972907008:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828118.200613:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828118.200624:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff9c1a7895de00 x1874668972907008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828118.200640:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972907008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828118.200651:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828118.200654:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828118.200657:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972907008:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828118.200668:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972907136:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828118.200701:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828118.200723:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828118/real 1787828118] req@ffff9c1a7895d2c0 x1874668972907136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828118.200748:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972907136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828118.200778:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828118.200782:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828118.200787:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972907136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828118.711455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828118.711539:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828118.967550:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828119.735492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828119.735532:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828119.991556:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828120.759459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828120.759459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828121.015453:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828121.783462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828121.783506:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828122.039533:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828122.807462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828122.807466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828123.063596:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828123.319542:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828123.319554:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828123.319560:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828123.319572:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4573 00000100:00080000:1.0:1787828123.319587:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828123.319596:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828123.319650:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972907264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828123.319690:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828123.319695:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828123.319698:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828123.319703:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4573 00000100:00080000:1.0:1787828123.319707:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828123.319713:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828123.319726:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828123.319744:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972907392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828123.319758:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828123.319763:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828123.319765:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828123.319771:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4573 00000100:00080000:1.0:1787828123.319778:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828123.319781:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972907264:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828123.319788:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828123.319800:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828123.319815:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972907520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828123.319830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828123.319835:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828123.319837:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828123.319842:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4573 00000100:00080000:1.0:1787828123.319848:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828123.319853:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828123.319858:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828123.319874:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972907648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828123.319885:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828123.319897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828123.319928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828123.319931:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828123.319940:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4573 00000100:00080000:1.0:1787828123.319946:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828123.319951:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828123.319958:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828123.319990:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972907776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828123.319991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff9c1a78a79e00 x1874668972907264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828123.320036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828123.320043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828123.320046:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828123.320056:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4573 00000100:00100000:2.0:1787828123.320058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972907264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828123.320061:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828123.320070:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828123.320079:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828123.320098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972907904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828123.320104:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828123.320117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828123.320127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828123.320127:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828123.320135:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828123.320145:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4573 00000100:00100000:2.0:1787828123.320147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972907264:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828123.320150:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828123.320160:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828123.320171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828123.320181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972907392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828123.320190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972908032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828123.320236:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828123.320258:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff9c1a78a7b0c0 x1874668972907392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828123.320284:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972907392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828123.320295:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828123.320299:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828123.320303:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972907392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828123.320336:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972907520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828123.320407:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828123.320422:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff9c1a78a7bc00 x1874668972907520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828123.320438:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972907520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828123.320463:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828123.320468:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828123.320472:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972907520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828123.320480:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972907648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828123.320503:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828123.320516:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff9c1a78a78780 x1874668972907648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828123.320531:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972907648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828123.320550:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828123.320554:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828123.320558:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972907648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828123.320565:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972907776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828123.320584:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828123.320595:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff9c1a78a7a1c0 x1874668972907776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828123.320609:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972907776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828123.320619:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828123.320622:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828123.320626:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972907776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828123.320632:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972907904:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828123.320652:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828123.320663:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff9c1a78a783c0 x1874668972907904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828123.320677:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972907904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828123.320687:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828123.320691:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828123.320696:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972907904:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828123.320703:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972908032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828123.320721:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828123.320731:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828123/real 1787828123] req@ffff9c1a78a7a940 x1874668972908032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828123.320745:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972908032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828123.320754:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828123.320757:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828123.320761:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972908032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828123.831477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828123.831491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828124.087607:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828124.855458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828124.855470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828125.111533:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828125.879472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828125.879593:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828126.135543:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828126.903458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828126.903459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828127.159497:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828127.927473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828127.927481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828128.183441:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828128.439505:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828128.439522:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828128.439529:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828128.439545:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4578 00000100:00080000:0.0:1787828128.439563:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828128.439575:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828128.439647:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972908160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828128.439698:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828128.439704:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828128.439707:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828128.439712:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4578 00000100:00080000:0.0:1787828128.439718:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828128.439724:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828128.439737:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828128.439758:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972908288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828128.439766:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972908160:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828128.439792:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828128.439797:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828128.439800:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828128.439808:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4578 00000100:00080000:0.0:1787828128.439812:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828128.439817:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828128.439824:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828128.439842:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972908416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828128.439864:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828128.439894:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828128.439928:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828128.439936:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828128.439946:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4578 00000100:00080000:0.0:1787828128.439955:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828128.439964:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828128.439973:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff9c1a788612c0 x1874668972908160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828128.439978:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828128.440009:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972908160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828128.440017:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972908544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828128.440039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828128.440053:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828128.440055:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828128.440063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972908160:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828128.440069:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828128.440077:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828128.440087:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4578 00000100:00080000:0.0:1787828128.440095:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828128.440107:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828128.440109:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972908288:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787828128.440119:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828128.440151:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972908672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828128.440173:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828128.440175:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828128.440185:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828128.440190:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828128.440200:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4578 00000100:00100000:2.0:1787828128.440205:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff9c1a78863840 x1874668972908288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828128.440211:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828128.440229:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828128.440235:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972908288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828128.440244:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828128.440249:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828128.440257:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828128.440262:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972908288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828128.440277:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972908416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828128.440280:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972908800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828128.440302:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828128.440311:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828128.440317:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828128.440324:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828128.440331:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff9c1a78860b40 x1874668972908416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828128.440339:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4578 00000100:00100000:2.0:1787828128.440355:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972908416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828128.440403:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828128.440409:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828128.440417:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972908416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828128.440432:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972908544:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828128.440436:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828128.440453:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828128.440465:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828128.440488:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828128.440498:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972908928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828128.440521:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff9c1a78862d00 x1874668972908544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828128.440542:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972908544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828128.440557:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828128.440563:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828128.440567:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972908544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828128.440610:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972908672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828128.440642:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828128.440656:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff9c1a78860f00 x1874668972908672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828128.440689:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972908672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828128.440702:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828128.440707:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828128.440713:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972908672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828128.440722:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972908800:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828128.440763:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828128.440775:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff9c1a78861a40 x1874668972908800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828128.440797:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972908800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828128.440807:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828128.440810:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828128.440813:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972908800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828128.440819:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972908928:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828128.440836:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828128.440846:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828128/real 1787828128] req@ffff9c1a78862940 x1874668972908928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828128.440859:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972908928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828128.440894:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828128.440898:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828128.440902:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972908928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828128.951501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828128.951595:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828129.207504:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828129.975505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828129.975508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828130.231590:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828130.999493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828130.999493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828131.255569:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828132.023508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828132.023522:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828132.279633:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828133.047462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828133.047539:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828133.303515:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828133.559519:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828133.559534:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828133.559542:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828133.559560:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4583 00000100:00080000:3.0:1787828133.559576:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828133.559605:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828133.559667:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972909056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828133.559700:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828133.559706:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828133.559709:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828133.559714:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4583 00000100:00080000:3.0:1787828133.559721:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828133.559728:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828133.559742:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828133.559750:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972909056:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828133.559788:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972909184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828133.559811:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828133.559817:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828133.559822:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828133.559856:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4583 00000100:00080000:3.0:1787828133.559874:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828133.559883:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828133.559894:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828133.559913:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828133.559920:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972909312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828133.559941:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828133.559949:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828133.559953:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828133.559966:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4583 00000100:00080000:3.0:1787828133.559973:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828133.559981:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828133.559996:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828133.560015:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828133/real 1787828133] req@ffff9c1a7895f840 x1874668972909056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828133.560019:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972909440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828133.560068:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828133.560082:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828133.560084:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828133.560092:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4583 00000100:00080000:3.0:1787828133.560097:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828133.560100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972909056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828133.560108:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787828133.560171:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828133.560184:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828133.560192:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972909056:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787828133.560249:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828133.560262:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972909184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828133.560286:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972909568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828133.560302:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828133.560303:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828133.560310:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828133.560313:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828133.560321:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4583 00000100:00080000:3.0:1787828133.560327:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828133.560330:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828133/real 1787828133] req@ffff9c1a7895c000 x1874668972909184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828133.560343:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972909184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828133.560389:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828133.560393:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828133.560401:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972909184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828133.560411:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972909312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828133.560442:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828133.560454:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828133/real 1787828133] req@ffff9c1a7895c3c0 x1874668972909312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828133.560463:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972909312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828133.560469:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828133.560472:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828133.560475:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972909312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828133.560482:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972909440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828133.560498:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828133.560513:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828133/real 1787828133] req@ffff9c1a7895cf00 x1874668972909440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828133.560527:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972909440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828133.560534:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828133.560536:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828133.560539:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972909440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828133.560559:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972909568:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828133.560580:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787828133.560584:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828133.560598:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828133.560608:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828133/real 1787828133] req@ffff9c1a7895f480 x1874668972909568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828133.560629:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972909568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828133.560637:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828133.560639:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828133.560642:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972909568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828133.560700:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972909696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828133.560728:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828133.560733:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828133.560736:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828133.560740:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4583 00000100:00080000:3.0:1787828133.560747:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828133.560752:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828133.560760:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828133.560768:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972909696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828133.560801:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972909824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828133.560816:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828133.560858:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828133/real 1787828133] req@ffff9c1a7895e940 x1874668972909696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828133.560896:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972909696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828133.560908:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828133.560912:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828133.560916:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972909696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828133.560938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972909824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828133.560966:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828133.560980:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828133/real 1787828133] req@ffff9c1a7895fc00 x1874668972909824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828133.560995:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972909824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828133.561005:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828133.561009:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828133.561014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972909824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828134.071436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828134.071442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828134.327650:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828135.095518:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828135.095533:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828135.351490:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828136.119432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828136.119432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828136.375506:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828137.143507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828137.143517:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828137.399545:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828138.167474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828138.167474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828138.423611:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828138.679453:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828138.679463:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828138.679468:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828138.679478:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4589 00000100:00080000:3.0:1787828138.679489:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828138.679497:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828138.679555:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972909952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828138.679587:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828138.679593:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828138.679597:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828138.679603:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4589 00000100:00080000:3.0:1787828138.679609:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828138.679617:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828138.679654:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828138.679673:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972910080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828138.679680:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972909952:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828138.679688:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828138.679692:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828138.679696:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828138.679700:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4589 00000100:00080000:3.0:1787828138.679705:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828138.679710:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828138.679716:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828138.679728:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972910208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828138.679741:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828138.679745:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828138.679748:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828138.679752:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4589 00000800:00000100:0.0:1787828138.679756:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828138.679757:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828138.679762:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828138.679768:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828138.679780:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972910336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828138.679791:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828138.679813:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828138.679816:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828138.679820:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828138/real 1787828138] req@ffff9c1a7895e580 x1874668972909952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828138.679821:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4589 00000100:00080000:3.0:1787828138.679826:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828138.679831:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828138.679837:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828138.679845:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972909952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828138.679852:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972910464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828138.679862:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828138.679865:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828138.679867:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828138.679869:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828138.679871:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828138.679873:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972909952:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828138.679877:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4589 00000100:00080000:3.0:1787828138.679881:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828138.679885:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828138.679890:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828138.679903:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972910080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828138.679903:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972910592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828138.679916:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828138.679920:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828138.679923:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828138.679928:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4589 00000800:00000100:0.0:1787828138.679933:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828138.679933:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828138.679938:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828138.679944:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828138.679950:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828138/real 1787828138] req@ffff9c1a7895fc00 x1874668972910080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828138.679958:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972910720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828138.679969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972910080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828138.679982:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828138.679987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828138.679992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972910080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828138.680002:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972910208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828138.680030:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828138.680046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828138/real 1787828138] req@ffff9c1a7895e940 x1874668972910208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828138.680058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972910208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828138.680066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828138.680069:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828138.680072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972910208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828138.680078:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972910336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828138.680096:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828138.680114:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828138/real 1787828138] req@ffff9c1a7895f480 x1874668972910336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828138.680126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972910336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828138.680134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828138.680137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828138.680140:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972910336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828138.680145:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972910464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828138.680161:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828138.680170:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828138/real 1787828138] req@ffff9c1a7895cf00 x1874668972910464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828138.680181:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972910464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828138.680188:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828138.680191:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828138.680194:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972910464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828138.680226:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972910592:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828138.680248:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828138.680260:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828138/real 1787828138] req@ffff9c1a7895c3c0 x1874668972910592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828138.680272:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972910592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828138.680282:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828138.680286:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828138.680289:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972910592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828138.680295:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972910720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828138.680341:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828138.680398:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828138/real 1787828138] req@ffff9c1a7895c000 x1874668972910720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828138.680417:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972910720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828138.680425:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828138.680428:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828138.680430:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972910720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828139.191446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828139.191475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828139.447629:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828140.215436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828140.215489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828140.471518:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828141.239414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828141.239414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828141.495466:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828142.263458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828142.263528:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828142.519470:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828143.287442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828143.287477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828143.543523:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828143.799421:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828143.799434:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828143.799439:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828143.799452:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4594 00000100:00080000:3.0:1787828143.799465:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828143.799472:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828143.799519:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972910848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828143.799546:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828143.799550:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828143.799551:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828143.799555:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4594 00000100:00080000:3.0:1787828143.799558:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828143.799561:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828143.799570:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828143.799582:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972910976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828143.799592:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828143.799595:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828143.799597:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828143.799603:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972910848:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828143.799604:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4594 00000100:00080000:3.0:1787828143.799612:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828143.799616:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828143.799620:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828143.799632:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972911104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828143.799648:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828143.799651:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828143.799653:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828143.799656:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4594 00000100:00080000:3.0:1787828143.799659:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828143.799662:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828143.799666:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828143.799677:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972911232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828143.799681:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828143.799691:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828143.799696:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828143.799698:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828143.799703:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4594 00000100:00080000:3.0:1787828143.799706:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828143.799710:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828143.799713:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828143.799724:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972911360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828143.799730:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828143/real 1787828143] req@ffff9c1a7895f840 x1874668972910848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828143.799741:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828143.799751:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828143.799756:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828143.799780:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4594 00000100:00080000:3.0:1787828143.799787:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828143.799795:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828143.799796:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972910848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828143.799801:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828143.799814:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828143.799820:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972911488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828143.799822:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828143.799832:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972910848:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828143.799842:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828143.799846:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828143.799848:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828143.799853:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4594 00000100:00080000:3.0:1787828143.799857:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828143.799860:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828143.799864:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828143.799865:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972910976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828143.799877:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972911616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828143.799919:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828143.799939:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828143/real 1787828143] req@ffff9c1a7895d2c0 x1874668972910976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828143.799977:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972910976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828143.799992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828143.799998:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828143.800003:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972910976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828143.800014:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972911104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828143.800047:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828143.800067:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828143/real 1787828143] req@ffff9c1a7895de00 x1874668972911104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828143.800087:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972911104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828143.800100:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828143.800105:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828143.800111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972911104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828143.800121:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972911232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828143.800165:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828143.800183:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828143/real 1787828143] req@ffff9c1a7895c780 x1874668972911232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828143.800224:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972911232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828143.800236:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828143.800240:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828143.800244:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972911232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828143.800253:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972911360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828143.800278:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828143.800292:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828143/real 1787828143] req@ffff9c1a7895cb40 x1874668972911360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828143.800308:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972911360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828143.800320:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828143.800324:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828143.800328:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972911360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828143.800336:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972911488:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828143.800362:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828143.800412:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828143/real 1787828143] req@ffff9c1a7895f0c0 x1874668972911488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828143.800429:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972911488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828143.800448:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828143.800452:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828143.800456:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972911488:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828143.800482:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972911616:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828143.800511:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828143.800527:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828143/real 1787828143] req@ffff9c1a7895da40 x1874668972911616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828143.800543:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972911616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828143.800554:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828143.800558:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828143.800562:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972911616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828144.311480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828144.311507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828144.567593:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828145.335432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828145.335455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828145.591516:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828146.359468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828146.359487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828146.615571:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828147.383495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828147.383518:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828147.639604:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828148.407540:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828148.407542:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828148.663575:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828148.919533:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828148.919546:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828148.919552:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828148.919565:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4599 00000100:00080000:0.0:1787828148.919581:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828148.919590:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828148.919643:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972911744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828148.919685:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828148.919690:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828148.919692:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828148.919698:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4599 00000100:00080000:0.0:1787828148.919702:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828148.919706:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828148.919718:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828148.919758:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972911872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828148.919773:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828148.919778:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828148.919781:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828148.919787:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4599 00000100:00080000:0.0:1787828148.919791:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828148.919796:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828148.919802:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828148.919816:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972911744:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828148.919819:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972912000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828148.919843:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828148.919849:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828148.919852:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828148.919856:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4599 00000100:00080000:0.0:1787828148.919861:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828148.919865:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828148.919871:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828148.919886:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972912128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828148.919897:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828148.919901:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828148.919904:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828148.919908:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4599 00000100:00080000:0.0:1787828148.919914:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828148.919922:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828148.919926:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828148.919935:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828148.919952:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972912256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828148.919965:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828148.919969:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828148.919972:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828148.919978:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4599 00000100:00080000:0.0:1787828148.919983:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828148.919988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828148/real 1787828148] req@ffff9c1a788621c0 x1874668972911744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828148.919992:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828148.919999:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828148.920014:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972912384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828148.920062:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828148.920066:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828148.920069:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828148.920077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972911744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828148.920081:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4599 00000100:00080000:0.0:1787828148.920091:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828148.920102:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828148.920115:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828148.920119:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828148.920131:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972912512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828148.920132:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828148.920148:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972911744:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828148.920169:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972911872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828148.920203:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828148.920219:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828148/real 1787828148] req@ffff9c1a78860780 x1874668972911872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828148.920244:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972911872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828148.920254:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828148.920258:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828148.920263:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972911872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828148.920292:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972912000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828148.920312:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828148.920324:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828148/real 1787828148] req@ffff9c1a78861e00 x1874668972912000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828148.920338:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972912000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828148.920389:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828148.920393:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828148.920398:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972912000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828148.920406:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972912128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828148.920430:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828148.920445:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828148/real 1787828148] req@ffff9c1a78862580 x1874668972912128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828148.920459:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972912128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828148.920479:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828148.920483:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828148.920487:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972912128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828148.920493:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972912256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828148.920512:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828148.920522:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828148/real 1787828148] req@ffff9c1a78861680 x1874668972912256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828148.920536:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972912256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828148.920547:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828148.920550:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828148.920554:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972912256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828148.920560:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972912384:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828148.920579:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828148.920589:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828148/real 1787828148] req@ffff9c1a788630c0 x1874668972912384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828148.920602:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972912384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828148.920611:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828148.920615:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828148.920620:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972912384:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828148.920626:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972912512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828148.920644:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828148.920654:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828148/real 1787828148] req@ffff9c1a78863c00 x1874668972912512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828148.920668:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972912512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828148.920678:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828148.920681:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828148.920685:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972912512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828149.431502:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828149.431502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828149.687527:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828150.455457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828150.455463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828150.711607:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828151.479496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828151.479516:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828151.735598:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828152.503497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828152.503503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828152.759545:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828153.527498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828153.527499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828153.783504:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828154.039595:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828154.039608:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828154.039614:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828154.039627:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4604 00000100:00080000:3.0:1787828154.039641:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828154.039650:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828154.039738:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972912640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828154.039779:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828154.039785:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828154.039787:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828154.039793:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4604 00000100:00080000:3.0:1787828154.039798:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828154.039803:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828154.039818:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828154.039838:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972912768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828154.039854:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828154.039859:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828154.039862:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828154.039872:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972912640:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828154.039874:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4604 00000100:00080000:3.0:1787828154.039884:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828154.039891:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828154.039898:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828154.039918:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972912896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828154.039940:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828154.039947:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828154.039951:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828154.039958:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4604 00000100:00080000:3.0:1787828154.039967:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828154.039972:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828154.039983:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828154.039996:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828154.040024:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972913024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828154.040045:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828154.040051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff9c1a7895e1c0 x1874668972912640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828154.040055:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828154.040065:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828154.040084:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4604 00000100:00100000:2.0:1787828154.040094:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972912640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828154.040095:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828154.040114:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828154.040118:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828154.040128:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828154.040130:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828154.040143:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972912640:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828154.040161:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972913152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828154.040177:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828154.040181:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828154.040184:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828154.040190:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4604 00000100:00100000:2.0:1787828154.040196:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972912768:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828154.040200:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828154.040215:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828154.040228:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828154.040239:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828154.040259:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972913280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828154.040265:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff9c1a7895da40 x1874668972912768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.040318:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972912768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828154.040338:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828154.040342:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.040397:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828154.040405:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972912768:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828154.040409:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828154.040417:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828154.040426:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972912896:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828154.040429:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4604 00000100:00080000:3.0:1787828154.040443:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828154.040450:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828154.040459:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828154.040468:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828154.040485:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972913408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828154.040495:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff9c1a7895f0c0 x1874668972912896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.040533:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972912896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828154.040545:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.040552:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828154.040557:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972912896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828154.040565:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972913024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828154.040587:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828154.040600:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff9c1a7895cb40 x1874668972913024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.040614:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972913024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828154.040624:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.040627:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828154.040631:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972913024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828154.040638:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972913152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828154.040656:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828154.040667:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff9c1a7895c780 x1874668972913152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.040681:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972913152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828154.040719:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.040723:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828154.040727:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972913152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828154.040750:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972913280:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828154.040775:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828154.040788:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff9c1a7895de00 x1874668972913280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.040811:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972913280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828154.040822:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.040825:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828154.040829:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972913280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828154.040836:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972913408:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828154.040854:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828154.040865:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828154/real 1787828154] req@ffff9c1a7895d2c0 x1874668972913408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828154.040879:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972913408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828154.040889:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828154.040892:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828154.040895:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972913408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828154.551445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828154.551452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828154.807611:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828155.575468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828155.575505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828155.831653:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828156.599438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828156.599443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828156.855514:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828157.623476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828157.623494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828157.879553:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828158.647410:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828158.647415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828158.903677:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828159.159559:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828159.159570:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828159.159574:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828159.159585:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4609 00000100:00080000:3.0:1787828159.159611:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828159.159619:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828159.159686:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972913536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828159.159754:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828159.159759:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828159.159762:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828159.159768:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4609 00000100:00080000:3.0:1787828159.159773:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828159.159781:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828159.159795:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828159.159815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972913664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828159.159825:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972913536:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828159.159841:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828159.159847:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828159.159849:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828159.159855:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4609 00000100:00080000:3.0:1787828159.159859:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828159.159864:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828159.159871:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828159.159888:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972913792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828159.159900:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828159.159906:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828159.159909:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828159.159913:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4609 00000100:00080000:3.0:1787828159.159918:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828159.159926:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828159.159930:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828159.159934:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828159.159952:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972913920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828159.159965:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828159.159970:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828159.159972:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828159.159976:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4609 00000100:00080000:3.0:1787828159.159981:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828159.160004:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828159.160011:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828159.160027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff9c1a7895f840 x1874668972913536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828159.160029:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972914048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828159.160068:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828159.160075:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828159.160078:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828159.160101:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4609 00000100:00080000:3.0:1787828159.160106:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828159.160110:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972913536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828159.160115:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828159.160127:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828159.160134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828159.160143:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972914176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828159.160144:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828159.160155:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972913536:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828159.160188:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828159.160192:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828159.160195:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828159.160201:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4609 00000100:00080000:3.0:1787828159.160206:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828159.160210:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828159.160218:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828159.160225:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972913664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828159.160239:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972914304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828159.160272:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828159.160296:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff9c1a7895c000 x1874668972913664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828159.160313:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972913664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828159.160324:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828159.160328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828159.160354:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972913664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828159.160361:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972913792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828159.160379:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828159.160388:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff9c1a7895c3c0 x1874668972913792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828159.160400:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972913792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828159.160473:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828159.160477:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828159.160481:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972913792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828159.160488:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972913920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828159.160514:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828159.160528:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff9c1a7895cf00 x1874668972913920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828159.160542:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972913920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828159.160570:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828159.160574:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828159.160578:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972913920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828159.160584:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972914048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828159.160606:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828159.160617:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff9c1a7895f480 x1874668972914048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828159.160631:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972914048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828159.160641:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828159.160644:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828159.160649:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972914048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828159.160681:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972914176:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828159.160710:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828159.160724:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff9c1a7895e940 x1874668972914176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828159.160738:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972914176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828159.160749:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828159.160752:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828159.160756:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972914176:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828159.160780:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972914304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828159.160800:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828159.160811:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828159/real 1787828159] req@ffff9c1a7895fc00 x1874668972914304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828159.160826:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972914304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828159.160836:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828159.160839:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828159.160842:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972914304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828159.671457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828159.671458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828159.927595:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828160.695425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828160.695462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828160.951582:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828161.719443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828161.719443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828161.975588:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828162.743497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828162.743499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828162.999617:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828163.767439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828163.767439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828164.023505:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828164.279470:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828164.279486:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828164.279493:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828164.279509:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4614 00000100:00080000:3.0:1787828164.279525:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828164.279535:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828164.279581:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972914432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828164.279658:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828164.279663:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828164.279675:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828164.279681:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4614 00000100:00080000:3.0:1787828164.279687:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828164.279692:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828164.279705:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828164.279729:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972914560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828164.279732:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972914432:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828164.279759:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828164.279763:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828164.279766:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828164.279773:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4614 00000100:00080000:3.0:1787828164.279777:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828164.279782:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828164.279788:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828164.279812:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972914688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828164.279825:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828164.279829:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828164.279831:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828164.279836:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4614 00000100:00080000:3.0:1787828164.279842:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828164.279844:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828164.279851:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828164.279876:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828164.279892:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972914816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828164.279904:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828164.279908:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828164.279910:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828164.279916:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4614 00000100:00080000:3.0:1787828164.279921:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828164.279926:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828164.279932:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828164.279941:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff9c1a7895e580 x1874668972914432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828164.279951:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972914944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828164.279992:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828164.279996:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828164.279999:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828164.280006:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972914432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828164.280010:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4614 00000100:00080000:3.0:1787828164.280020:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828164.280032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828164.280042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828164.280047:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828164.280055:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972914432:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787828164.280059:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828164.280098:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972915072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828164.280112:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828164.280116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972914560:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828164.280118:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828164.280125:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828164.280132:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4614 00000100:00080000:3.0:1787828164.280138:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828164.280142:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828164.280149:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828164.280162:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828164.280176:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972915200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828164.280202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff9c1a7895fc00 x1874668972914560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828164.280229:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972914560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828164.280241:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828164.280247:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828164.280252:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972914560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828164.280260:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972914688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828164.280284:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828164.280313:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff9c1a7895e940 x1874668972914688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828164.280327:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972914688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828164.280386:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828164.280390:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828164.280394:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972914688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828164.280401:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972914816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828164.280447:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828164.280474:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff9c1a7895f480 x1874668972914816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828164.280489:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972914816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828164.280500:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828164.280504:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828164.280507:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972914816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828164.280514:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972914944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828164.280534:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828164.280545:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff9c1a7895cf00 x1874668972914944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828164.280567:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972914944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828164.280577:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828164.280580:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828164.280583:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972914944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828164.280605:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972915072:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828164.280671:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828164.280686:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff9c1a7895c3c0 x1874668972915072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828164.280700:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972915072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828164.280709:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828164.280712:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828164.280716:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972915072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828164.280722:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972915200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828164.280740:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828164.280750:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828164/real 1787828164] req@ffff9c1a7895c000 x1874668972915200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828164.280762:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972915200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828164.280777:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828164.280780:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828164.280783:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972915200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828164.791453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828164.791453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828165.047573:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828165.815430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828165.815432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828166.071589:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828166.839506:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828166.839526:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828167.095506:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828167.863454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828167.863461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828168.119575:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828168.887438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828168.887438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828169.143549:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828169.399478:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828169.399490:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828169.399497:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828169.399509:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4619 00000100:00080000:3.0:1787828169.399522:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828169.399531:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828169.399579:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972915328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828169.399637:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828169.399642:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828169.399645:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828169.399650:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4619 00000100:00080000:3.0:1787828169.399655:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828169.399659:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828169.399681:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828169.399699:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972915456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828169.399717:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972915328:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828169.399718:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828169.399734:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828169.399739:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828169.399745:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4619 00000100:00080000:3.0:1787828169.399749:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828169.399754:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828169.399761:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828169.399778:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972915584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828169.399800:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828169.399805:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828169.399808:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828169.399812:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4619 00000100:00080000:3.0:1787828169.399827:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828169.399832:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828169.399838:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828169.399862:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972915712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828169.399874:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828169.399878:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828169.399881:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828169.399886:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4619 00000100:00080000:3.0:1787828169.399892:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828169.399898:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828169.399906:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828169.399918:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972915456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828169.399927:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972915840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828169.399945:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828169.399950:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828169.399952:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828169.399958:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4619 00000100:00080000:3.0:1787828169.399974:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828169.399989:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972915584:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828169.399991:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828169.400009:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828169.400032:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972915968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828169.400045:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828169.400048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972915712:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828169.400057:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828169.400063:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828169.400071:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4619 00000100:00080000:3.0:1787828169.400078:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828169.400084:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828169.400091:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828169.400105:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972915840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828169.400116:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972916096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828169.400149:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972915968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828169.400187:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972916096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828169.911469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828169.911524:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828170.167498:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828170.935481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828170.935481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828171.191529:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828171.959470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828171.959471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828172.215646:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828172.471837:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787828172.471954:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828172.471988:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828172.472066:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828172.472107:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828172.472134:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828172.472152:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828172.472181:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828172.472203:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787828172.472206:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828172] req@ffff9c1a7895d2c0 x1874668972915456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828172.472284:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972915456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828172.472307:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828172.472317:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828172.472325:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972915456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828172.472339:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828172] req@ffff9c1a7895de00 x1874668972915584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828172.472436:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972915584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828172.472453:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828172.472464:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828172.472472:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972915584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828172.472490:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828172] req@ffff9c1a7895c780 x1874668972915712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828172.472522:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972915712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828172.472538:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828172.472546:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828172.472552:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972915712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828172.472591:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828172] req@ffff9c1a7895f840 x1874668972915328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828172.472617:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972915328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828172.472631:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828172.472637:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828172.472642:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972915328:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828172.472691:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828172] req@ffff9c1a7895da40 x1874668972916096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828172.472737:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972916096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828172.472755:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828172.472769:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828172.472776:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972916096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828172.472789:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828172] req@ffff9c1a7895cb40 x1874668972915840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828172.472815:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972915840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828172.472831:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828172.472837:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828172.472843:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972915840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828172.472856:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828169/real 1787828172] req@ffff9c1a7895f0c0 x1874668972915968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828172.472882:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972915968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828172.472899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828172.472905:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828172.472912:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972915968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828172.983477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828172.983478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828173.239580:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828174.007413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828174.007461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828174.263476:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828174.519483:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828174.519521:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828174.519527:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828174.519542:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4624 00000100:00080000:3.0:1787828174.519576:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828174.519587:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828174.519645:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972916224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828174.519689:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828174.519693:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828174.519696:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828174.519701:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4624 00000100:00080000:3.0:1787828174.519706:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828174.519711:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828174.519723:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828174.519746:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972916352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828174.519748:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972916224:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828174.519769:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828174.519774:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828174.519776:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828174.519781:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4624 00000100:00080000:3.0:1787828174.519785:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828174.519788:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828174.519794:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828174.519807:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972916480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828174.519819:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828174.519823:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828174.519825:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828174.519832:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4624 00000800:00000100:2.0:1787828174.519833:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828174.519841:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828174.519848:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828174.519854:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828174.519881:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972916608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828174.519890:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828174/real 1787828174] req@ffff9c1a7895e1c0 x1874668972916224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828174.519896:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828174.519900:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828174.519902:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828174.519906:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4624 00000100:00080000:3.0:1787828174.519928:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828174.519935:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828174.519953:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828174.519967:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972916224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828174.519973:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972916736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828174.520007:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828174.520008:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828174.520018:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828174.520021:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828174.520023:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828174.520032:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972916224:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828174.520034:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4624 00000100:00080000:3.0:1787828174.520057:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828174.520065:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828174.520074:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972916352:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787828174.520074:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828174.520103:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972916864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828174.520123:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828174.520132:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828174.520140:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828174.520143:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828174.520151:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4624 00000100:00100000:2.0:1787828174.520154:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828174/real 1787828174] req@ffff9c1a7895f0c0 x1874668972916352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828174.520158:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828174.520170:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828174.520182:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828174.520187:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972916352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828174.520204:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972916992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828174.520207:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828174.520220:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828174.520233:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972916352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828174.520247:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972916480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828174.520280:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828174.520297:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828174/real 1787828174] req@ffff9c1a7895cb40 x1874668972916480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828174.520314:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972916480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828174.520327:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828174.520332:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828174.520336:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972916480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828174.520379:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972916608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828174.520411:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828174.520428:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828174/real 1787828174] req@ffff9c1a7895da40 x1874668972916608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828174.520448:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972916608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828174.520460:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828174.520464:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828174.520470:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972916608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828174.520478:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972916736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828174.520505:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828174.520521:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828174/real 1787828174] req@ffff9c1a7895f840 x1874668972916736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828174.520539:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972916736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828174.520580:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828174.520586:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828174.520591:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972916736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828174.520620:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972916864:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828174.520650:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828174.520666:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828174/real 1787828174] req@ffff9c1a7895c780 x1874668972916864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828174.520685:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972916864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828174.520698:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828174.520703:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828174.520708:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972916864:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828174.520717:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972916992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828174.520740:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828174.520760:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828174/real 1787828174] req@ffff9c1a7895de00 x1874668972916992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828174.520780:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972916992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828174.520792:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828174.520796:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828174.520801:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972916992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828175.031486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828175.031486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828175.287620:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828176.055426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828176.055436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828176.311606:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828177.079477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828177.079481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828177.335681:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828178.103432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828178.103452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828178.359491:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828179.127506:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828179.127508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828179.383636:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828179.639612:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828179.639623:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828179.639629:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828179.639640:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4629 00000100:00080000:0.0:1787828179.639653:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828179.639662:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828179.639722:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972917120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828179.639757:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828179.639762:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828179.639764:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828179.639769:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4629 00000100:00080000:0.0:1787828179.639773:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828179.639777:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828179.639789:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828179.639804:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972917248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828179.639816:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828179.639821:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828179.639823:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828179.639831:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4629 00000100:00080000:0.0:1787828179.639836:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828179.639840:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828179.639846:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828179.639860:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972917376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828179.639875:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828179.639885:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828179.639890:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972917120:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828179.639891:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828179.639914:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4629 00000100:00080000:0.0:1787828179.639918:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828179.639922:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828179.639927:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828179.639943:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972917504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828179.639954:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828179.639958:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828179.639960:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828179.639964:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4629 00000100:00080000:0.0:1787828179.639967:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828179.639971:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828179.639975:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828179.639988:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972917632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828179.639997:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828179.640002:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828179.640004:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828179.640008:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4629 00000100:00080000:0.0:1787828179.640013:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828179.640028:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828179.640034:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787828179.640051:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828179.640083:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972917760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828179.640104:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828179.640113:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828179/real 1787828179] req@ffff9c1a78860000 x1874668972917120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828179.640115:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828179.640129:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828179.640153:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4629 00000100:00100000:2.0:1787828179.640165:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972917120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828179.640169:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828179.640194:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828179.640200:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828179.640210:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787828179.640211:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828179.640223:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972917120:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828179.640239:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972917248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828179.640248:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972917888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828179.640291:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828179.640311:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828179/real 1787828179] req@ffff9c1a78863c00 x1874668972917248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828179.640328:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972917248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828179.640399:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828179.640405:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828179.640410:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972917248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828179.640461:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972917376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828179.640502:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828179.640551:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828179/real 1787828179] req@ffff9c1a788630c0 x1874668972917376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828179.640578:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972917376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828179.640595:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828179.640601:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828179.640606:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972917376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828179.640615:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972917504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828179.640647:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828179.640663:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828179/real 1787828179] req@ffff9c1a78861680 x1874668972917504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828179.640687:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972917504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828179.640698:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828179.640702:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828179.640705:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972917504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828179.640713:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972917632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828179.640732:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828179.640743:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828179/real 1787828179] req@ffff9c1a78862580 x1874668972917632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828179.640757:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972917632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828179.640767:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828179.640770:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828179.640774:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972917632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828179.640781:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972917760:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828179.640801:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828179.640811:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828179/real 1787828179] req@ffff9c1a78861e00 x1874668972917760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828179.640825:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972917760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828179.640842:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828179.640846:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828179.640850:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972917760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828179.640856:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972917888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828179.640874:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828179.640885:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828179/real 1787828179] req@ffff9c1a78860780 x1874668972917888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828179.640898:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972917888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828179.640907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828179.640911:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828179.640914:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972917888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828180.151538:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828180.151538:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828180.407540:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828181.175499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828181.175517:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828181.431588:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828182.199461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828182.199552:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828182.455587:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828183.223439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828183.223470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828183.479618:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828184.247459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828184.247499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828184.503533:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828184.759583:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828184.759594:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828184.759600:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828184.759611:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4635 00000100:00080000:2.0:1787828184.759623:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828184.759632:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828184.759680:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972918016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828184.759715:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828184.759720:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828184.759722:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828184.759726:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4635 00000100:00080000:2.0:1787828184.759730:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828184.759735:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828184.759747:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828184.759760:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972918144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828184.759771:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972918016:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828184.759778:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828184.759784:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828184.759785:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828184.759790:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4635 00000100:00080000:2.0:1787828184.759793:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828184.759797:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828184.759802:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828184.759816:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972918272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828184.759828:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828184.759831:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828184.759833:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828184.759838:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4635 00000100:00080000:2.0:1787828184.759843:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828184.759846:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828184.759852:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828184.759858:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828184.759871:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972918400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828184.759882:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828184.759886:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828184.759888:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828184.759893:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4635 00000100:00080000:2.0:1787828184.759896:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828184.759900:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828184.759906:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828184.759925:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828184/real 1787828184] req@ffff9c1a49d8c780 x1874668972918016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828184.759929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972918528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828184.759979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828184.759985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828184.759991:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972918016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828184.759992:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828184.760007:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4635 00000100:00080000:2.0:1787828184.760018:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828184.760020:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828184.760026:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787828184.760032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:2.0:1787828184.760038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828184.760039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972918016:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828184.760063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972918656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828184.760077:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972918144:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828184.760080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828184.760087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828184.760089:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828184.760095:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4635 00000100:00080000:2.0:1787828184.760100:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828184.760104:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828184.760110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828184.760117:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828184.760131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972918784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828184.760139:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828184/real 1787828184] req@ffff9c1a49d8de00 x1874668972918144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828184.760160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972918144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828184.760171:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828184.760175:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828184.760179:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972918144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828184.760185:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972918272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828184.760204:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828184.760214:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828184/real 1787828184] req@ffff9c1a49d8c000 x1874668972918272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828184.760225:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972918272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828184.760240:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828184.760244:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828184.760247:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972918272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828184.760252:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972918400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828184.760267:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828184.760275:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828184/real 1787828184] req@ffff9c1a49d8da40 x1874668972918400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828184.760286:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972918400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828184.760295:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828184.760297:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828184.760300:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972918400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828184.760306:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972918528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828184.760320:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828184.760329:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828184/real 1787828184] req@ffff9c1a49d8cf00 x1874668972918528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828184.760340:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972918528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828184.760375:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828184.760378:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828184.760398:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972918528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828184.760424:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972918656:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828184.760452:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828184.760467:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828184/real 1787828184] req@ffff9c1a49d8f0c0 x1874668972918656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828184.760509:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972918656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828184.760520:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828184.760540:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828184.760544:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972918656:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828184.760551:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972918784:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828184.760572:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828184.760583:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828184/real 1787828184] req@ffff9c1a49d8cb40 x1874668972918784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828184.760595:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972918784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828184.760603:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828184.760606:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828184.760609:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972918784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828185.271429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828185.271451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828185.527456:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828186.295444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828186.295485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828186.551579:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828187.319426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828187.319454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828187.575514:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828188.343472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828188.343505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828188.599467:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828189.367484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828189.367499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828189.623614:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828189.879512:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828189.879529:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828189.879536:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828189.879551:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4640 00000100:00080000:3.0:1787828189.879566:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828189.879575:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828189.879623:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972918912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828189.879671:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828189.879677:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828189.879681:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828189.879688:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4640 00000100:00080000:3.0:1787828189.879693:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828189.879699:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828189.879712:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828189.879741:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972918912:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828189.879742:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972919040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828189.879770:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828189.879773:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828189.879776:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828189.879781:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4640 00000100:00080000:3.0:1787828189.879786:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828189.879790:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828189.879796:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828189.879811:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972919168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828189.879826:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828189.879830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828189.879833:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828189.879836:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4640 00000100:00080000:3.0:1787828189.879852:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828189.879857:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828189.879863:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828189.879872:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828189.879885:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972919296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828189.879901:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828189.879905:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828189.879908:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828189.879913:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4640 00000100:00080000:3.0:1787828189.879917:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828189.879921:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828189.879927:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828189.879942:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972919424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828189.879952:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828189/real 1787828189] req@ffff9c1a7895d2c0 x1874668972918912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828189.879960:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828189.879975:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828189.879982:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828189.879991:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4640 00000100:00080000:3.0:1787828189.880005:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828189.880011:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828189.880017:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972918912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828189.880021:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828189.880044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972919552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828189.880045:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828189.880071:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828189.880071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828189.880085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828189.880087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972918912:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828189.880091:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828189.880102:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4640 00000100:00080000:3.0:1787828189.880107:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828189.880112:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828189.880128:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972919040:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787828189.880130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828189.880157:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972919680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828189.880180:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828189.880202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828189/real 1787828189] req@ffff9c1a7895c000 x1874668972919040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828189.880227:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972919040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828189.880239:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828189.880243:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828189.880247:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972919040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828189.880254:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972919168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828189.880276:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828189.880303:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828189/real 1787828189] req@ffff9c1a7895c3c0 x1874668972919168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828189.880327:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972919168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828189.880343:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828189.880412:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828189.880418:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972919168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828189.880427:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972919296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828189.880491:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828189.880508:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828189/real 1787828189] req@ffff9c1a7895cf00 x1874668972919296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828189.880524:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972919296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828189.880536:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828189.880540:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828189.880544:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972919296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828189.880553:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972919424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828189.880574:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828189.880601:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828189/real 1787828189] req@ffff9c1a7895f480 x1874668972919424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828189.880625:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972919424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828189.880636:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828189.880640:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828189.880644:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972919424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828189.880651:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972919552:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828189.880675:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828189.880687:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828189/real 1787828189] req@ffff9c1a7895e940 x1874668972919552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828189.880701:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972919552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828189.880711:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828189.880714:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828189.880718:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972919552:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828189.880760:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972919680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828189.880797:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828189.880820:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828189/real 1787828189] req@ffff9c1a7895fc00 x1874668972919680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828189.880845:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972919680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828189.880875:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828189.880882:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828189.880889:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972919680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828190.391504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828190.391560:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828190.647597:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828191.415455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828191.415457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828191.671520:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828192.439468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828192.439496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828192.695476:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828193.463413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828193.463413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828193.719486:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828194.487461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828194.487461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828194.743516:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828194.999472:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828194.999480:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828194.999484:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828194.999499:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4645 00000100:00080000:1.0:1787828194.999510:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828194.999516:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828194.999551:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972919808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828194.999573:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828194.999576:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828194.999578:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828194.999581:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4645 00000100:00080000:1.0:1787828194.999583:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828194.999585:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828194.999593:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828194.999603:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972919936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828194.999610:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828194.999612:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828194.999614:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828194.999616:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4645 00000100:00080000:1.0:1787828194.999618:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828194.999627:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972919808:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828194.999628:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828194.999635:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828194.999643:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972920064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828194.999652:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828194.999655:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828194.999656:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828194.999659:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4645 00000100:00080000:1.0:1787828194.999662:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828194.999665:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828194.999668:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828194.999677:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972920192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828194.999683:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828194.999686:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828194.999687:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828194.999692:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828194.999692:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4645 00000100:00080000:1.0:1787828194.999697:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828194.999702:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828194.999706:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828194.999715:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972920320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828194.999722:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828194.999724:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828194.999726:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828194.999729:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4645 00000100:00100000:0.0:1787828194.999733:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828194/real 1787828194] req@ffff9c1a78a78f00 x1874668972919808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828194.999734:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828194.999741:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828194.999748:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828194.999760:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972920448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828194.999766:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972919808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828194.999772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828194.999779:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828194.999783:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828194.999788:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4645 00000100:00080000:0.0:1787828194.999791:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828194.999792:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828194.999798:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787828194.999801:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:1.0:1787828194.999802:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828194.999808:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972919808:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828194.999814:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972920576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828194.999820:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972919936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828194.999848:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828194.999860:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828194/real 1787828194] req@ffff9c1a78a7a580 x1874668972919936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828194.999869:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972919936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828194.999875:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828194.999877:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828194.999880:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972919936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828194.999898:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972920064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828194.999910:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828194.999916:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828194/real 1787828194] req@ffff9c1a78a78000 x1874668972920064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828194.999924:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972920064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828194.999929:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828194.999931:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828194.999933:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972920064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828194.999937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972920192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828194.999946:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828194.999951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828194/real 1787828194] req@ffff9c1a78a7b480 x1874668972920192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828194.999958:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972920192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828194.999964:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828194.999965:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828194.999967:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972920192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828194.999971:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972920320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828194.999981:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828194.999987:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828194/real 1787828194] req@ffff9c1a78a792c0 x1874668972920320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828194.999994:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972920320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828194.999999:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828195.000000:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828195.000002:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972920320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828195.000005:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972920448:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828195.000016:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828195.000022:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828194/real 1787828195] req@ffff9c1a78a7b840 x1874668972920448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828195.000029:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972920448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828195.000036:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828195.000039:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828195.000042:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972920448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828195.000047:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972920576:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828195.000066:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828195.000077:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828194/real 1787828195] req@ffff9c1a78a79a40 x1874668972920576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828195.000090:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972920576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828195.000096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828195.000098:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828195.000100:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972920576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828195.511424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828195.511427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828195.767473:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828196.535412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828196.535443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828196.791515:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828197.559417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828197.559420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828197.815475:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828198.583429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828198.583433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828198.839523:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828199.607429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828199.607440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828199.863435:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828200.119467:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828200.119475:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828200.119479:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828200.119487:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4650 00000100:00080000:3.0:1787828200.119495:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828200.119502:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828200.119536:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972920704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828200.119561:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828200.119566:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828200.119567:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828200.119570:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4650 00000100:00080000:3.0:1787828200.119572:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828200.119575:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828200.119583:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828200.119594:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972920832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828200.119601:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828200.119604:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828200.119605:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828200.119607:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4650 00000100:00080000:3.0:1787828200.119609:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828200.119611:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828200.119614:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828200.119624:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972920960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828200.119628:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972920704:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828200.119637:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828200.119640:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828200.119641:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828200.119644:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4650 00000100:00080000:3.0:1787828200.119646:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828200.119649:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828200.119654:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828200.119669:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972921088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828200.119675:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828200.119677:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828200.119679:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828200.119681:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4650 00000100:00080000:3.0:1787828200.119683:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828200.119685:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828200.119690:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828200.119698:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972921216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828200.119705:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828200.119708:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828200.119709:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828200.119711:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4650 00000100:00080000:3.0:1787828200.119714:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828200.119717:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787828200.119726:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828200.119729:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828200.119742:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972921344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828200.119751:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828200.119753:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828200.119755:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828200.119759:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4650 00000100:00080000:3.0:1787828200.119762:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828200.119766:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828200.119771:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828200.119777:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff9c1a7895e580 x1874668972920704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828200.119782:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972921472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828200.119854:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972920704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.119870:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.119876:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828200.119879:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972920704:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828200.119886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972920832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828200.119909:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828200.119919:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff9c1a7895fc00 x1874668972920832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828200.119928:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972920832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.119933:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.119936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828200.119939:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972920832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828200.119962:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972920960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828200.119973:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828200.119980:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff9c1a7895e940 x1874668972920960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828200.119988:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972920960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.119993:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.119995:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828200.119997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972920960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828200.120001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972921088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828200.120011:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828200.120017:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff9c1a7895f480 x1874668972921088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828200.120024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972921088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.120030:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.120032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828200.120034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972921088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828200.120038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972921216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828200.120048:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828200.120054:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff9c1a7895cf00 x1874668972921216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828200.120061:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972921216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.120066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.120068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828200.120070:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972921216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828200.120074:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972921344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828200.120085:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828200.120090:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff9c1a7895c3c0 x1874668972921344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828200.120097:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972921344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.120103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.120104:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828200.120106:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972921344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828200.120110:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972921472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828200.120119:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828200.120125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828200/real 1787828200] req@ffff9c1a7895c000 x1874668972921472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828200.120138:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972921472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828200.120143:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828200.120145:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828200.120147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972921472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828200.631429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828200.631435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828200.887517:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828201.655429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828201.655436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828201.911486:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828202.679442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828202.679443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828202.935506:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828203.703468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828203.703468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828203.959463:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828204.727418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828204.727425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828204.983498:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828205.239454:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828205.239465:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828205.239470:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.239481:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4655 00000100:00080000:1.0:1787828205.239493:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.239502:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828205.239541:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972921600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828205.239572:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828205.239575:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828205.239577:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.239583:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4655 00000100:00080000:1.0:1787828205.239589:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.239596:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828205.239607:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828205.239618:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972921600:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828205.239620:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972921728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828205.239634:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828205.239638:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828205.239640:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.239645:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4655 00000100:00080000:1.0:1787828205.239648:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.239651:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828205.239655:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828205.239666:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972921856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828205.239677:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828205.239686:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828205.239688:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.239694:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4655 00000800:00000100:3.0:1787828205.239700:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828205.239703:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.239708:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828205.239714:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828205.239725:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972921984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828205.239735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828205.239740:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828205.239741:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.239745:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4655 00000100:00080000:1.0:1787828205.239747:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.239751:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828205.239755:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828205.239763:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972922112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828205.239770:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828205.239773:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828205.239775:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.239779:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4655 00000100:00080000:1.0:1787828205.239783:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.239787:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828205.239792:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828205.239802:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972922240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828205.239810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828205.239812:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828205.239814:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828205.239819:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4655 00000100:00080000:1.0:1787828205.239823:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828205.239826:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828205.239830:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828205.239837:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972922368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828205.239990:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff9c1a78a79a40 x1874668972921600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828205.240055:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972921600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828205.240069:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828205.240080:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828205.240086:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972921600:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828205.240113:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972921728:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828205.240137:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828205.240147:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff9c1a78a7b840 x1874668972921728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828205.240156:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972921728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828205.240162:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828205.240164:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828205.240166:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972921728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828205.240170:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972921856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828205.240181:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828205.240187:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff9c1a78a792c0 x1874668972921856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828205.240195:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972921856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828205.240200:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828205.240202:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828205.240204:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972921856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828205.240208:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972921984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828205.240218:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828205.240224:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff9c1a78a7b480 x1874668972921984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828205.240236:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972921984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828205.240242:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828205.240244:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828205.240245:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972921984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828205.240249:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972922112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828205.240259:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828205.240264:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff9c1a78a78000 x1874668972922112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828205.240272:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972922112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828205.240277:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828205.240278:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828205.240280:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972922112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828205.240284:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972922240:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828205.240295:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828205.240301:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff9c1a78a7a580 x1874668972922240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828205.240308:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972922240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828205.240317:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828205.240320:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828205.240322:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972922240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828205.240325:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972922368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828205.240373:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828205.240392:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828205/real 1787828205] req@ffff9c1a78a78f00 x1874668972922368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828205.240402:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972922368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828205.240409:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828205.240413:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828205.240416:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972922368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828205.751495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828205.751495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828206.007564:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828206.775410:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828206.775420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828207.031451:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828207.799414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828207.799414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828208.055474:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828208.823413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828208.823416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828209.079487:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828209.847426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828209.847426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828210.103470:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828210.359432:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828210.359440:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828210.359444:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828210.359453:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4660 00000100:00080000:1.0:1787828210.359461:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.359468:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828210.359500:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972922496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828210.359523:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828210.359526:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828210.359528:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828210.359531:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4660 00000100:00080000:1.0:1787828210.359533:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.359536:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828210.359544:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828210.359553:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972922624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828210.359563:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828210.359566:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828210.359567:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828210.359572:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4660 00000100:00080000:1.0:1787828210.359579:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828210.359580:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972922496:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828210.359583:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828210.359597:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828210.359605:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972922752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828210.359616:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828210.359618:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828210.359619:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828210.359623:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4660 00000100:00080000:1.0:1787828210.359626:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.359628:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828210.359631:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828210.359639:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972922880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828210.359644:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828210.359646:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828210.359647:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828210.359650:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4660 00000100:00080000:1.0:1787828210.359652:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.359655:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828210.359659:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828210.359667:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972923008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828210.359672:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828210.359683:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828210.359686:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828210.359687:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828210.359692:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4660 00000100:00080000:1.0:1787828210.359695:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.359698:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828210.359704:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828210.359713:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972923136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828210.359722:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828210.359724:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828210.359726:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828210.359730:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4660 00000100:00100000:3.0:1787828210.359731:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff9c1a78a7a940 x1874668972922496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828210.359733:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828210.359744:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828210.359756:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828210.359775:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972923264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828210.359782:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972922496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828210.359797:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828210.359801:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828210.359809:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972922496:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828210.359836:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972922624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828210.359861:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828210.359875:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff9c1a78a783c0 x1874668972922624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828210.359889:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972922624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828210.359899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828210.359902:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828210.359904:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972922624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828210.359910:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972922752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828210.359927:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828210.359935:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff9c1a78a7a1c0 x1874668972922752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828210.359943:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972922752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828210.359954:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828210.359956:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828210.359958:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972922752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828210.359962:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972922880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828210.359973:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828210.359979:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff9c1a78a78780 x1874668972922880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828210.359987:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972922880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828210.359992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828210.359994:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828210.359996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972922880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828210.360000:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972923008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828210.360010:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828210.360016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff9c1a78a7bc00 x1874668972923008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828210.360024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972923008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828210.360029:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828210.360031:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828210.360033:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972923008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828210.360037:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972923136:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828210.360048:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828210.360053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff9c1a78a7b0c0 x1874668972923136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828210.360060:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972923136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828210.360064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828210.360066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828210.360068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972923136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828210.360072:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972923264:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828210.360082:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828210.360088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828210/real 1787828210] req@ffff9c1a78a79e00 x1874668972923264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828210.360095:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972923264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828210.360101:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828210.360102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828210.360104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972923264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828210.871405:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828210.871415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828211.127468:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828211.895437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828211.895437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828212.151537:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828212.919440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828212.919448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828213.175503:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828213.943454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828213.943454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828214.199482:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828214.967484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828214.967538:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828215.223470:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828215.479447:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828215.479456:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828215.479460:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828215.479468:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4665 00000100:00080000:0.0:1787828215.479478:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828215.479484:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828215.479519:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972923392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828215.479542:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828215.479545:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828215.479547:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828215.479550:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4665 00000100:00080000:0.0:1787828215.479553:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828215.479557:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828215.479566:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828215.479578:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972923392:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828215.479579:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972923520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828215.479593:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828215.479596:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828215.479597:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828215.479600:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4665 00000100:00080000:0.0:1787828215.479602:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828215.479604:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828215.479607:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828215.479617:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972923648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828215.479624:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828215.479626:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828215.479627:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828215.479630:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4665 00000100:00080000:0.0:1787828215.479633:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828215.479637:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828215.479641:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828215.479642:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828215.479651:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972923776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828215.479659:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828215.479662:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828215.479664:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828215.479666:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4665 00000100:00080000:0.0:1787828215.479669:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828215.479671:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828215.479675:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828215.479689:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828215/real 1787828215] req@ffff9c1a788621c0 x1874668972923392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828215.479690:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972923904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828215.479727:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828215.479731:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787828215.479735:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972923392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828215.479736:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828215.479747:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4665 00000100:00080000:3.0:1787828215.479751:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828215.479754:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828215.479757:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828215.479766:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828215.479766:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972923392:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787828215.479779:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828215.479795:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972924032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828215.479796:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972923520:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828215.479809:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828215.479814:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828215.479816:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828215.479820:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4665 00000100:00080000:0.0:1787828215.479823:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828215.479826:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828215.479831:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828215.479836:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828215.479843:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972924160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828215.479856:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828215/real 1787828215] req@ffff9c1a78862940 x1874668972923520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828215.479870:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972923520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828215.479877:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828215.479879:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828215.479882:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972923520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828215.479887:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972923648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828215.479902:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828215.479909:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828215/real 1787828215] req@ffff9c1a78861a40 x1874668972923648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828215.479918:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972923648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828215.479923:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828215.479925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828215.479927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972923648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828215.479931:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972923776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828215.479949:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828215.479956:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828215/real 1787828215] req@ffff9c1a78860f00 x1874668972923776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828215.479964:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972923776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828215.479970:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828215.479972:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828215.479974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972923776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828215.479978:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972923904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828215.479989:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828215.479995:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828215/real 1787828215] req@ffff9c1a78862d00 x1874668972923904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828215.480007:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972923904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828215.480013:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828215.480015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828215.480017:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972923904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828215.480029:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972924032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828215.480041:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828215.480047:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828215/real 1787828215] req@ffff9c1a78860b40 x1874668972924032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828215.480055:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972924032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828215.480062:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828215.480065:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828215.480068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972924032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828215.480075:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972924160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828215.480094:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828215.480114:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828215/real 1787828215] req@ffff9c1a78863840 x1874668972924160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828215.480126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972924160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828215.480137:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828215.480139:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828215.480141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972924160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828215.991430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828215.991430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828216.247507:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828217.015429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828217.015429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828217.271448:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828218.039452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828218.039452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828218.295502:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828219.063427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828219.063428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828219.319521:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828220.087427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828220.087431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828220.343463:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828220.599470:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828220.599479:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828220.599482:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828220.599490:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4670 00000100:00080000:0.0:1787828220.599500:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828220.599517:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828220.599550:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972924288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828220.599574:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828220.599576:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828220.599578:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828220.599580:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4670 00000100:00080000:0.0:1787828220.599583:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828220.599586:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828220.599593:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828220.599602:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972924416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828220.599610:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828220.599612:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828220.599614:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828220.599616:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4670 00000100:00080000:0.0:1787828220.599618:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828220.599621:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828220.599623:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828220.599629:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972924544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828220.599637:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828220.599640:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787828220.599640:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972924288:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828220.599643:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828220.599649:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4670 00000100:00080000:0.0:1787828220.599653:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828220.599655:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828220.599657:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828220.599675:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972924672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828220.599681:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828220.599683:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828220.599684:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828220.599686:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4670 00000100:00080000:0.0:1787828220.599690:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828220.599692:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828220.599695:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828220.599703:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972924800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828220.599708:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828220.599711:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828220.599713:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828220.599715:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4670 00000100:00080000:0.0:1787828220.599718:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828220.599720:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828220.599723:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828220.599728:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828220.599732:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972924928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828220.599751:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828220.599753:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828220.599755:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828220.599757:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4670 00000100:00080000:0.0:1787828220.599759:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828220.599762:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828220.599764:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828220.599771:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972925056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828220.599790:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828220/real 1787828220] req@ffff9c1a788612c0 x1874668972924288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828220.599824:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972924288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828220.599838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828220.599844:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828220.599850:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972924288:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828220.599860:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972924416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828220.599888:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828220.599904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828220/real 1787828220] req@ffff9c1a78863840 x1874668972924416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828220.599926:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972924416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828220.599937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828220.599941:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828220.599945:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972924416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828220.599969:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972924544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828220.599993:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828220.600005:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828220/real 1787828220] req@ffff9c1a78860b40 x1874668972924544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828220.600019:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972924544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828220.600028:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828220.600032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828220.600036:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972924544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828220.600043:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972924672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828220.600065:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828220.600077:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828220/real 1787828220] req@ffff9c1a78862d00 x1874668972924672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828220.600091:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972924672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828220.600106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828220.600110:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828220.600113:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972924672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828220.600120:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972924800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828220.600141:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828220.600154:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828220/real 1787828220] req@ffff9c1a78860f00 x1874668972924800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828220.600167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972924800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828220.600176:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828220.600180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828220.600184:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972924800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828220.600190:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972924928:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828220.600213:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828220.600243:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828220/real 1787828220] req@ffff9c1a78861a40 x1874668972924928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828220.600259:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972924928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828220.600267:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828220.600270:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828220.600274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972924928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828220.600281:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972925056:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828220.600303:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828220.600315:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828220/real 1787828220] req@ffff9c1a78862940 x1874668972925056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828220.600328:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972925056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828220.600338:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828220.600341:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828220.600375:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972925056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828221.111418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828221.111418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828221.367447:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828222.135431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828222.135440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828222.391491:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828223.159471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828223.159471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828223.415487:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828224.183431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828224.183462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828224.439569:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828225.207454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828225.207477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828225.463547:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828225.719417:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828225.719427:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828225.719431:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828225.719442:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4676 00000100:00080000:0.0:1787828225.719453:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828225.719461:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828225.719500:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972925184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828225.719531:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828225.719540:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828225.719542:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828225.719547:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4676 00000100:00080000:0.0:1787828225.719552:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828225.719557:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828225.719568:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828225.719579:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972925184:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828225.719586:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972925312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828225.719599:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828225.719604:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828225.719606:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828225.719611:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4676 00000100:00080000:0.0:1787828225.719615:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828225.719620:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828225.719625:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828225.719646:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828225.719648:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972925440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828225.719663:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828225.719667:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828225.719670:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828225.719674:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4676 00000100:00080000:0.0:1787828225.719679:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828225.719684:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828225.719690:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828225/real 1787828225] req@ffff9c1a788621c0 x1874668972925184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828225.719692:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828225.719710:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972925184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828225.719714:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972925568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828225.719726:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828225.719732:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828225.719734:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828225.719740:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787828225.719741:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972925184:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828225.719746:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828225.719753:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4676 00000100:00080000:0.0:1787828225.719757:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828225.719761:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828225.719766:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972925312:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787828225.719770:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828225.719788:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972925696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828225.719801:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828225.719803:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828225.719808:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828225.719810:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828225.719814:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828225/real 1787828225] req@ffff9c1a78860780 x1874668972925312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828225.719819:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4676 00000100:00080000:0.0:1787828225.719828:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828225.719833:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972925312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828225.719838:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787828225.719844:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828225.719848:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828225.719848:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828225.719854:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972925312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828225.719859:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972925440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828225.719866:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972925824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828225.719876:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828225.719882:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828225.719887:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828225/real 1787828225] req@ffff9c1a78861e00 x1874668972925440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828225.719888:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828225.719893:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828225.719903:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4676 00000100:00100000:3.0:1787828225.719907:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972925440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828225.719909:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828225.719919:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787828225.719919:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828225.719926:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787828225.719930:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828225.719938:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972925440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828225.719946:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972925952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828225.719946:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972925568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828225.719965:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828225.719974:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828225/real 1787828225] req@ffff9c1a78862580 x1874668972925568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828225.719983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972925568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828225.719989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828225.719991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828225.719993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972925568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828225.720003:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972925696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828225.720013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828225.720019:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828225/real 1787828225] req@ffff9c1a78861680 x1874668972925696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828225.720026:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972925696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828225.720032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828225.720033:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828225.720035:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972925696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828225.720039:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972925824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828225.720049:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828225.720054:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828225/real 1787828225] req@ffff9c1a788630c0 x1874668972925824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828225.720066:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972925824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828225.720071:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828225.720073:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828225.720074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972925824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828225.720078:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972925952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828225.720087:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828225.720092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828225/real 1787828225] req@ffff9c1a78863c00 x1874668972925952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828225.720099:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972925952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828225.720104:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828225.720106:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828225.720107:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972925952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828226.231425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828226.231432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828226.487526:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828227.255441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828227.255452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828227.511463:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828228.279413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828228.279434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828228.535435:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828229.303425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828229.303438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828229.559465:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828230.327421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828230.327434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828230.583489:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828230.839465:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828230.839475:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828230.839479:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828230.839490:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4681 00000100:00080000:1.0:1787828230.839503:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828230.839512:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828230.839564:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972926080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828230.839591:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828230.839594:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828230.839596:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828230.839599:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4681 00000100:00080000:1.0:1787828230.839602:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828230.839605:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828230.839615:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828230.839624:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972926208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828230.839632:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828230.839634:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828230.839635:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828230.839638:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4681 00000100:00080000:1.0:1787828230.839642:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828230.839645:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828230.839648:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828230.839655:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972926336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828230.839662:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972926080:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828230.839666:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828230.839670:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828230.839671:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828230.839675:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4681 00000100:00080000:1.0:1787828230.839677:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828230.839680:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828230.839682:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828230.839690:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972926464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828230.839695:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828230.839698:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828230.839700:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828230.839702:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4681 00000100:00080000:1.0:1787828230.839705:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828230.839707:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828230.839710:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828230.839717:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972926592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828230.839727:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828230.839729:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828230.839730:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828230.839733:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4681 00000100:00080000:1.0:1787828230.839735:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828230.839739:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828230.839741:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828230.839746:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828230.839754:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972926720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828230.839760:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828230.839763:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828230.839764:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828230.839766:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4681 00000100:00080000:1.0:1787828230.839768:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828230.839770:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828230.839773:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828230.839782:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972926848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828230.839788:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828230/real 1787828230] req@ffff9c1a78a79e00 x1874668972926080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828230.839813:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972926080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828230.839822:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828230.839826:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828230.839829:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972926080:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828230.839835:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972926208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828230.839853:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828230.839861:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828230/real 1787828230] req@ffff9c1a78a7b0c0 x1874668972926208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828230.839870:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972926208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828230.839876:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828230.839878:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828230.839880:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972926208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828230.839902:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972926336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828230.839913:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828230.839919:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828230/real 1787828230] req@ffff9c1a78a7bc00 x1874668972926336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828230.839926:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972926336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828230.839937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828230.839939:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828230.839941:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972926336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828230.839944:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972926464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828230.839954:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828230.839959:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828230/real 1787828230] req@ffff9c1a78a78780 x1874668972926464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828230.839967:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972926464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828230.839973:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828230.839974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828230.839976:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972926464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828230.839980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972926592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828230.839990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828230.839995:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828230/real 1787828230] req@ffff9c1a78a7a1c0 x1874668972926592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828230.840003:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972926592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828230.840008:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828230.840009:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828230.840012:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972926592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828230.840016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972926720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828230.840027:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828230.840032:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828230/real 1787828230] req@ffff9c1a78a783c0 x1874668972926720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828230.840039:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972926720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828230.840045:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828230.840047:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828230.840048:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972926720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828230.840052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972926848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828230.840062:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828230.840067:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828230/real 1787828230] req@ffff9c1a78a7a940 x1874668972926848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828230.840075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972926848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828230.840080:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828230.840081:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828230.840083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972926848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828231.351466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828231.351466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828231.607497:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828232.375418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828232.375426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828232.631459:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828233.399437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828233.399437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828233.655605:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828234.423449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828234.423451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828234.679475:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828235.447484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828235.447493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828235.703546:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828235.959499:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828235.959513:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828235.959519:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828235.959532:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4686 00000100:00080000:1.0:1787828235.959549:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828235.959560:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828235.959614:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972926976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828235.959650:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828235.959655:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828235.959658:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828235.959674:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4686 00000100:00080000:1.0:1787828235.959679:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828235.959684:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828235.959698:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828235.959716:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972927104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828235.959733:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828235.959741:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787828235.959744:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972926976:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828235.959745:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828235.959754:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4686 00000100:00080000:1.0:1787828235.959763:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828235.959773:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828235.959779:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828235.959793:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972927232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828235.959807:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828235.959813:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828235.959816:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828235.959821:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4686 00000100:00080000:1.0:1787828235.959828:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828235.959832:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828235.959838:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828235.959862:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972927360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828235.959882:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828235.959888:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828235.959893:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828235.959903:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4686 00000100:00080000:1.0:1787828235.959911:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828235.959928:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972927104:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828235.959944:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828235.959959:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828235.959986:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972927488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828235.960001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972927232:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828235.960008:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828235.960017:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828235.960021:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828235.960029:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4686 00000100:00080000:1.0:1787828235.960038:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828235.960044:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828235.960053:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972927360:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787828235.960057:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828235.960087:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972927616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828235.960102:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828235.960108:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828235.960111:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828235.960143:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972927488:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828235.960145:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4686 00000100:00080000:1.0:1787828235.960160:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828235.960187:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828235.960196:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828235.960225:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972927744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828235.960225:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972927616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828235.960287:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972927744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828236.471441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828236.471444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828236.727479:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828237.495467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828237.495507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828237.751497:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828238.519433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828238.519445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828238.775542:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828239.031653:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787828239.031696:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828239.031712:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828239.031748:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828239.031756:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828239.031764:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828239.031774:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828239.031783:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828239.031795:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787828239.031814:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828235/real 1787828239] req@ffff9c1a78a792c0 x1874668972927488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828239.031863:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972927488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828239.031880:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828239.031887:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828239.031894:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972927488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828239.031906:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828235/real 1787828239] req@ffff9c1a78a7b840 x1874668972927616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828239.031920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972927616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828239.031928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828239.031933:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828239.031938:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972927616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828239.031957:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828235/real 1787828239] req@ffff9c1a78a7a580 x1874668972927104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828239.031973:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972927104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828239.031984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828239.031989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828239.031993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972927104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828239.032002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828235/real 1787828239] req@ffff9c1a78a78000 x1874668972927232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828239.032018:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972927232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828239.032028:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828239.032032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828239.032037:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972927232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828239.032046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828235/real 1787828239] req@ffff9c1a78a7b480 x1874668972927360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828239.032059:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972927360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828239.032069:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828239.032073:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828239.032077:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972927360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828239.032086:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828235/real 1787828239] req@ffff9c1a78a78f00 x1874668972926976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828239.032124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972926976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828239.032136:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828239.032140:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828239.032145:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972926976:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828239.032186:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828235/real 1787828239] req@ffff9c1a78a79a40 x1874668972927744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828239.032203:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972927744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828239.032214:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828239.032219:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828239.032224:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972927744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828239.543451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828239.543548:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828239.799556:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828240.567419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828240.567427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828240.823513:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828241.079445:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.079454:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828241.079457:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.079466:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4691 00000100:00080000:3.0:1787828241.079476:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828241.079481:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828241.079514:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972927872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828241.079537:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.079542:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828241.079545:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.079550:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4691 00000100:00080000:3.0:1787828241.079554:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828241.079559:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828241.079570:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828241.079571:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972927872:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828241.079593:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972928000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828241.079605:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.079608:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828241.079611:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.079615:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4691 00000100:00080000:3.0:1787828241.079621:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828241.079634:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828241.079640:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828241.079648:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828241.079657:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972928128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828241.079669:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.079673:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828241.079675:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.079680:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4691 00000100:00080000:3.0:1787828241.079685:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828241.079695:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828241] req@ffff9c1a7895d2c0 x1874668972927872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828241.079698:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828241.079709:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828241.079730:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972928256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828241.079736:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972927872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828241.079748:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.079756:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828241.079760:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828241.079766:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828241.079768:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4691 00000100:00080000:3.0:1787828241.079776:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828241.079779:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828241.079784:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828241.079790:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972927872:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787828241.079793:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828241.079810:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972928384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828241.079824:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828241.079825:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972928000:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828241.079833:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828241.079837:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.079842:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4691 00000100:00080000:3.0:1787828241.079849:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828241.079854:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828241.079861:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828241.079865:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828241.079879:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972928512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828241.079887:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828241] req@ffff9c1a7895de00 x1874668972928000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828241.079895:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828241.079905:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828241.079909:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828241.079916:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4691 00000100:00100000:1.0:1787828241.079917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972928000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828241.079922:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828241.079931:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828241.079937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828241.079939:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828241.079946:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828241.079952:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972928000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828241.079957:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972928640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828241.079965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972928128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828241.079996:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828241.080011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828241] req@ffff9c1a7895c780 x1874668972928128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828241.080024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972928128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828241.080033:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828241.080037:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828241.080040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972928128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828241.080047:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972928256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828241.080074:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828241.080103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828241] req@ffff9c1a7895f840 x1874668972928256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828241.080118:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972928256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828241.080127:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828241.080131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828241.080134:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972928256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828241.080153:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972928384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828241.080177:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828241.080190:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828241] req@ffff9c1a7895da40 x1874668972928384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828241.080203:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972928384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828241.080212:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828241.080215:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828241.080219:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972928384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828241.080226:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972928512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828241.080248:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828241.080261:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828241] req@ffff9c1a7895cb40 x1874668972928512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828241.080275:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972928512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828241.080291:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828241.080294:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828241.080298:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972928512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828241.080305:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972928640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828241.080327:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828241.080338:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828241/real 1787828241] req@ffff9c1a7895f0c0 x1874668972928640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828241.080395:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972928640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828241.080405:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828241.080409:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828241.080413:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972928640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828241.591437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828241.591442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828241.847587:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828242.615473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828242.615478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828242.871459:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828243.639428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828243.639431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828243.895475:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828244.663434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828244.663434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828244.919522:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828245.687484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828245.687489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828245.943561:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828246.199456:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828246.199470:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828246.199475:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828246.199487:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4696 00000100:00080000:3.0:1787828246.199502:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828246.199525:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828246.199576:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972928768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828246.199617:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828246.199622:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828246.199625:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828246.199630:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4696 00000100:00080000:3.0:1787828246.199635:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828246.199639:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828246.199651:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828246.199667:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972928896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828246.199681:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828246.199686:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828246.199689:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828246.199693:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4696 00000100:00080000:3.0:1787828246.199697:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828246.199702:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828246.199707:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828246.199719:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972929024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828246.199736:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828246.199742:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972928768:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828246.199742:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828246.199749:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828246.199763:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4696 00000100:00080000:3.0:1787828246.199767:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828246.199771:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828246.199777:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828246.199793:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972929152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828246.199805:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828246.199809:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828246.199812:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828246.199817:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4696 00000100:00080000:3.0:1787828246.199822:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828246.199826:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828246.199833:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828246.199848:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828246.199858:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972929280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828246.199880:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828246.199887:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828246.199891:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828246.199900:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4696 00000100:00100000:1.0:1787828246.199910:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff9c1a7895e1c0 x1874668972928768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828246.199912:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828246.199929:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828246.199945:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828246.199952:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972928768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.199971:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828246.199973:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972929408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828246.199981:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828246.199998:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972928768:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828246.200002:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828246.200010:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828246.200014:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828246.200016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972928896:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828246.200025:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4696 00000100:00080000:3.0:1787828246.200033:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828246.200039:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828246.200045:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828246.200084:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828246.200105:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972929536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828246.200110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff9c1a7895f0c0 x1874668972928896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828246.200166:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972928896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.200179:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828246.200185:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828246.200190:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972928896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828246.200222:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972929024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828246.200248:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828246.200261:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff9c1a7895cb40 x1874668972929024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828246.200276:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972929024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.200286:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828246.200290:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828246.200293:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972929024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828246.200301:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972929152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828246.200319:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828246.200331:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff9c1a7895da40 x1874668972929152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828246.200391:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972929152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.200408:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828246.200415:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828246.200422:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972929152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828246.200434:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972929280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828246.200474:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828246.200498:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff9c1a7895f840 x1874668972929280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828246.200523:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972929280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.200534:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828246.200538:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828246.200542:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972929280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828246.200551:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972929408:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828246.200592:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828246.200606:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff9c1a7895c780 x1874668972929408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828246.200620:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972929408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.200636:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828246.200642:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828246.200649:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972929408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828246.200660:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972929536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828246.200697:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828246.200720:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828246/real 1787828246] req@ffff9c1a7895de00 x1874668972929536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828246.200743:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972929536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828246.200754:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828246.200758:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828246.200762:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972929536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828246.711479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828246.711479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828246.967653:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828247.735419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828247.735426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828247.991522:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828248.759441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828248.759442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828249.015536:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828249.783445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828249.783477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828250.039436:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828250.807430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828250.807435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828251.063569:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828251.319586:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828251.319597:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828251.319603:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828251.319616:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4701 00000100:00080000:1.0:1787828251.319629:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828251.319638:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828251.319708:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972929664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828251.319759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828251.319764:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828251.319766:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828251.319771:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4701 00000100:00080000:1.0:1787828251.319776:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828251.319781:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828251.319793:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828251.319811:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972929792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828251.319826:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828251.319830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828251.319834:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828251.319839:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972929664:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828251.319843:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4701 00000100:00080000:1.0:1787828251.319853:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828251.319859:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828251.319865:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828251.319881:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972929920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828251.319912:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828251.319917:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828251.319919:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828251.319923:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4701 00000100:00080000:1.0:1787828251.319929:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828251.319933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828251.319940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828251.319953:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828251.319958:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972930048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828251.319985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828251.319996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828251.319999:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828251.320004:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4701 00000100:00080000:1.0:1787828251.320008:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828251.320045:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828251.320045:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff9c1a78a79a40 x1874668972929664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828251.320054:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828251.320073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972930176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828251.320086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828251.320090:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828251.320092:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828251.320101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972929664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828251.320102:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4701 00000100:00080000:1.0:1787828251.320110:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828251.320119:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828251.320130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828251.320145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972930304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828251.320197:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828251.320200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828251.320209:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828251.320213:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828251.320215:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828251.320230:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4701 00000100:00100000:3.0:1787828251.320232:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972929664:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828251.320235:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828251.320246:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828251.320252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828251.320265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972930432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828251.320296:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972929792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828251.320342:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828251.320395:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff9c1a78a78f00 x1874668972929792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828251.320416:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972929792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828251.320434:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828251.320441:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828251.320448:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972929792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828251.320459:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972929920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828251.320501:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828251.320525:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff9c1a78a7b480 x1874668972929920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828251.320543:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972929920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828251.320554:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828251.320558:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828251.320562:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972929920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828251.320571:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972930048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828251.320605:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828251.320627:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff9c1a78a78000 x1874668972930048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828251.320649:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972930048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828251.320667:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828251.320673:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828251.320680:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972930048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828251.320690:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972930176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828251.320728:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828251.320750:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff9c1a78a7a580 x1874668972930176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828251.320775:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972930176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828251.320792:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828251.320799:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828251.320806:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972930176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828251.320818:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972930304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828251.320854:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828251.320872:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff9c1a78a7b840 x1874668972930304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828251.320891:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972930304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828251.320920:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828251.320925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828251.320930:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972930304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828251.320954:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972930432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828251.320979:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828251.320991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828251/real 1787828251] req@ffff9c1a78a792c0 x1874668972930432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828251.321004:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972930432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828251.321038:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828251.321045:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828251.321049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972930432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828251.831427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828251.831448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828252.087457:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828252.855443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828252.855443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828253.111488:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828253.879423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828253.879428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828254.135491:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828254.903488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828254.903489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828255.159469:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828255.927462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828255.927475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828256.183539:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828256.439477:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828256.439491:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828256.439496:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828256.439509:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4706 00000100:00080000:2.0:1787828256.439524:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828256.439533:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828256.439585:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972930560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828256.439625:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828256.439630:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828256.439632:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828256.439639:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4706 00000100:00080000:2.0:1787828256.439644:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828256.439649:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828256.439662:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828256.439681:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972930688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828256.439683:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972930560:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828256.439721:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828256.439726:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828256.439730:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828256.439737:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4706 00000100:00080000:2.0:1787828256.439741:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828256.439747:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828256.439755:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828256.439769:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972930816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828256.439783:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828256.439798:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828256.439817:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828256.439820:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828256.439824:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4706 00000100:00080000:2.0:1787828256.439830:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828256.439833:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828256.439839:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828256.439851:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972930944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828256.439858:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828256.439863:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787828256.439864:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff9c1a49d8e1c0 x1874668972930560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828256.439866:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828256.439879:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4706 00000100:00080000:2.0:1787828256.439888:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828256.439898:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828256.439919:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972930560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828256.439923:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828256.439951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828256.439953:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972931072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828256.439975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828256.439993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972930560:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828256.440021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828256.440023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828256.440024:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828256.440029:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4706 00000100:00080000:2.0:1787828256.440033:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828256.440035:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972930688:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828256.440042:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828256.440049:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828256.440061:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972931200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828256.440068:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828256.440084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828256.440088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff9c1a49d8d680 x1874668972930688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828256.440089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828256.440093:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828256.440108:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4706 00000100:00080000:2.0:1787828256.440115:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828256.440123:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828256.440127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828256.440128:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972930688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828256.440144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828256.440146:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972931328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828256.440156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828256.440170:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972930688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828256.440194:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972930816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828256.440211:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828256.440218:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff9c1a49d8ed00 x1874668972930816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828256.440226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972930816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828256.440232:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828256.440235:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828256.440237:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972930816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828256.440242:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972930944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828256.440254:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828256.440260:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff9c1a49d8fc00 x1874668972930944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828256.440268:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972930944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828256.440273:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828256.440275:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828256.440277:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972930944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828256.440281:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972931072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828256.440291:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828256.440297:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff9c1a49d8e580 x1874668972931072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828256.440304:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972931072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828256.440310:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828256.440311:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828256.440313:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972931072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828256.440335:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972931200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828256.440374:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828256.440382:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff9c1a49d8c3c0 x1874668972931200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828256.440391:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972931200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828256.440396:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828256.440399:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828256.440400:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972931200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828256.440404:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972931328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828256.440415:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828256.440421:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828256/real 1787828256] req@ffff9c1a49d8e1c0 x1874668972931328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828256.440429:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972931328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828256.440434:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828256.440436:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828256.440438:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972931328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828256.951478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828256.951478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828257.207586:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828257.975486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828257.975499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828258.231607:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828258.999442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828258.999490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828259.255509:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828260.023459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828260.023467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828260.279579:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828261.047447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828261.047468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828261.303570:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828261.559474:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828261.559487:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828261.559493:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828261.559507:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4711 00000100:00080000:3.0:1787828261.559523:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828261.559533:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828261.559586:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972931456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828261.559647:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828261.559652:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828261.559654:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828261.559660:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4711 00000100:00080000:3.0:1787828261.559664:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828261.559669:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828261.559694:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828261.559717:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972931456:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828261.559717:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972931584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828261.559736:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828261.559746:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828261.559748:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828261.559753:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4711 00000100:00080000:3.0:1787828261.559757:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828261.559760:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828261.559766:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828261.559780:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972931712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828261.559793:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828261.559797:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828261.559800:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828261.559805:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4711 00000100:00080000:3.0:1787828261.559809:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828261.559813:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828261.559819:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828261.559830:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972931840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828261.559857:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828261.559861:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828261.559863:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828261.559867:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828261.559873:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4711 00000100:00080000:3.0:1787828261.559877:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828261.559881:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828261.559887:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828261.559899:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972931968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828261.559908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828261.559911:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828261.559913:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828261.559918:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4711 00000100:00080000:3.0:1787828261.559921:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828261.559925:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828261.559931:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828261.559939:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828261/real 1787828261] req@ffff9c1a7895d2c0 x1874668972931456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828261.559968:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972932096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828261.559998:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828261.560002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828261.560004:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828261.560012:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972931456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828261.560012:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4711 00000100:00080000:3.0:1787828261.560023:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828261.560032:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828261.560039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828261.560040:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828261.560052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828261.560063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972931456:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828261.560064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972932224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828261.560104:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972931584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828261.560137:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828261.560153:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828261/real 1787828261] req@ffff9c1a7895c000 x1874668972931584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828261.560166:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972931584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828261.560175:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828261.560178:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828261.560182:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972931584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828261.560188:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972931712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828261.560205:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828261.560214:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828261/real 1787828261] req@ffff9c1a7895c3c0 x1874668972931712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828261.560226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972931712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828261.560233:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828261.560236:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828261.560239:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972931712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828261.560244:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972931840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828261.560260:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828261.560269:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828261/real 1787828261] req@ffff9c1a7895cf00 x1874668972931840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828261.560279:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972931840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828261.560289:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828261.560294:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828261.560299:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972931840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828261.560309:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972931968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828261.560337:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828261.560393:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828261/real 1787828261] req@ffff9c1a7895f480 x1874668972931968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828261.560413:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972931968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828261.560427:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828261.560432:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828261.560438:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972931968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828261.560449:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972932096:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828261.560480:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828261.560493:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828261/real 1787828261] req@ffff9c1a7895e940 x1874668972932096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828261.560505:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972932096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828261.560513:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828261.560517:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828261.560521:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972932096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828261.560527:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972932224:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828261.560543:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828261.560552:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828261/real 1787828261] req@ffff9c1a7895fc00 x1874668972932224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828261.560563:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972932224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828261.560571:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828261.560573:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828261.560576:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972932224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828262.071467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828262.071467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828262.327508:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828263.095438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828263.095438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828263.351507:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828264.119441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828264.119441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828264.375576:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828265.143439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828265.143474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828265.399646:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828266.167421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828266.167431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828266.423499:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828266.679462:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828266.679470:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828266.679474:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828266.679484:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4717 00000100:00080000:1.0:1787828266.679498:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828266.679506:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828266.679550:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972932352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828266.679577:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828266.679580:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828266.679582:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828266.679586:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4717 00000100:00080000:1.0:1787828266.679590:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828266.679595:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828266.679604:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828266.679615:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972932352:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828266.679620:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972932480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828266.679631:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828266.679634:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828266.679635:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828266.679639:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4717 00000100:00080000:1.0:1787828266.679641:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828266.679644:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828266.679647:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828266.679657:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972932608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828266.679665:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828266.679667:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828266.679669:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828266.679673:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4717 00000100:00080000:1.0:1787828266.679676:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828266.679679:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787828266.679683:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:1.0:1787828266.679684:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828266.679700:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972932736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828266.679712:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828266.679716:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828266.679719:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828266.679724:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828266/real 1787828266] req@ffff9c1a78a7a940 x1874668972932352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828266.679729:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4717 00000100:00080000:1.0:1787828266.679751:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828266.679758:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972932352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828266.679760:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828266.679769:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828266.679775:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828266.679781:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828266.679787:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972932864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828266.679788:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972932352:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828266.679813:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828266.679816:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828266.679819:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828266.679823:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4717 00000100:00080000:1.0:1787828266.679827:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828266.679832:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972932480:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828266.679833:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828266.679842:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828266.679855:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972932992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828266.679865:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828266.679866:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828266.679872:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828266.679874:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828266.679879:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4717 00000100:00100000:3.0:1787828266.679884:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828266/real 1787828266] req@ffff9c1a78a783c0 x1874668972932480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828266.679886:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828266.679895:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828266.679904:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828266.679923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972932480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828266.679931:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972933120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828266.679945:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828266.679953:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828266.679958:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972932480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828266.679965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972932608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828266.679985:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828266.679994:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828266/real 1787828266] req@ffff9c1a78a7a1c0 x1874668972932608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828266.680014:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972932608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828266.680019:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828266.680022:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828266.680024:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972932608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828266.680028:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972932736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828266.680040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828266.680047:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828266/real 1787828266] req@ffff9c1a78a78780 x1874668972932736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828266.680055:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972932736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828266.680061:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828266.680063:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828266.680065:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972932736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828266.680069:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972932864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828266.680086:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828266.680092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828266/real 1787828266] req@ffff9c1a78a7bc00 x1874668972932864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828266.680100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972932864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828266.680105:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828266.680107:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828266.680109:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972932864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828266.680121:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972932992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828266.680133:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828266.680149:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828266/real 1787828266] req@ffff9c1a78a7b0c0 x1874668972932992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828266.680157:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972932992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828266.680163:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828266.680165:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828266.680167:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972932992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828266.680172:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972933120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828266.680182:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828266.680188:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828266/real 1787828266] req@ffff9c1a78a79e00 x1874668972933120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828266.680196:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972933120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828266.680202:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828266.680214:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828266.680216:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972933120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828267.191409:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828267.191413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828267.447490:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828268.215528:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828268.215528:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828268.471593:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828269.239466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828269.239511:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828269.495696:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828270.263504:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828270.263505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828270.519578:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828271.287452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828271.287480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828271.543429:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828271.799476:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828271.799506:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828271.799511:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828271.799524:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4722 00000100:00080000:1.0:1787828271.799537:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828271.799546:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828271.799593:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972933248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828271.799626:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828271.799631:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828271.799634:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828271.799639:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4722 00000100:00080000:1.0:1787828271.799644:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828271.799650:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828271.799662:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828271.799680:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972933248:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828271.799685:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972933376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828271.799705:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828271.799711:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828271.799713:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828271.799719:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4722 00000100:00080000:1.0:1787828271.799725:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828271.799732:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828271.799740:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828271.799752:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828271.799761:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972933504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828271.799794:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828271.799800:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828271.799804:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828271.799810:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4722 00000100:00080000:1.0:1787828271.799817:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828271.799820:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828271/real 1787828271] req@ffff9c1a78a79e00 x1874668972933248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828271.799829:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828271.799844:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828271.799850:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972933248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828271.799865:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828271.799888:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972933632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828271.799893:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828271.799904:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972933248:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828271.799938:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828271.799944:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828271.799947:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828271.799954:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4722 00000100:00100000:3.0:1787828271.799959:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972933376:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828271.799962:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828271.799974:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828271.799984:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828271.799998:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828271.800007:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972933760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828271.800015:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828271/real 1787828271] req@ffff9c1a78a7b0c0 x1874668972933376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828271.800043:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828271.800048:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972933376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828271.800051:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828271.800058:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828271.800065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828271.800071:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4722 00000100:00080000:3.0:1787828271.800075:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828271.800082:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972933376:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828271.800083:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828271.800091:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972933504:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828271.800096:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828271.800107:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828271.800115:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828271.800129:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972933888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828271.800131:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828271/real 1787828271] req@ffff9c1a78a7bc00 x1874668972933504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828271.800172:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828271.800179:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972933504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828271.800180:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828271.800186:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828271.800197:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4722 00000100:00080000:3.0:1787828271.800204:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828271.800207:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828271.800216:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828271.800220:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828271.800228:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972933504:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787828271.800232:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828271.800243:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972933632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828271.800256:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972934016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828271.800282:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828271.800298:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828271/real 1787828271] req@ffff9c1a78a78780 x1874668972933632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828271.800315:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972933632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828271.800323:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828271.800325:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828271.800328:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972933632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828271.800368:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972933760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828271.800386:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828271.800396:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828271/real 1787828271] req@ffff9c1a78a7a1c0 x1874668972933760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828271.800406:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972933760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828271.800413:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828271.800416:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828271.800418:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972933760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828271.800423:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972933888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828271.800437:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828271.800444:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828271/real 1787828271] req@ffff9c1a78a783c0 x1874668972933888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828271.800453:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972933888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828271.800465:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828271.800468:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828271.800470:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972933888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828271.800475:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972934016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828271.800487:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828271.800494:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828271/real 1787828271] req@ffff9c1a78a7a940 x1874668972934016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828271.800506:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972934016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828271.800517:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828271.800521:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828271.800525:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972934016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828272.311516:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828272.311523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828272.567586:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828273.335477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828273.335529:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828273.591586:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828274.359456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828274.359466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828274.615599:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828275.383455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828275.383486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828275.639542:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828276.407421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828276.407423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828276.663502:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828276.919464:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828276.919472:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828276.919476:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828276.919484:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4727 00000100:00080000:3.0:1787828276.919493:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828276.919499:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828276.919531:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972934144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828276.919560:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828276.919563:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828276.919564:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828276.919568:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4727 00000100:00080000:3.0:1787828276.919570:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828276.919573:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828276.919581:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828276.919598:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972934144:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828276.919598:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972934272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828276.919614:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828276.919616:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828276.919618:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828276.919622:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4727 00000100:00080000:3.0:1787828276.919624:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828276.919628:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828276.919631:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828276.919642:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972934400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828276.919649:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828276.919652:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828276.919653:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828276.919656:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4727 00000100:00080000:3.0:1787828276.919659:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828276.919662:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787828276.919667:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828276.919668:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828276.919679:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972934528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828276.919686:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828276.919688:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828276.919690:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828276.919693:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4727 00000100:00080000:3.0:1787828276.919696:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828276.919698:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828276.919702:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828276.919708:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828276/real 1787828276] req@ffff9c1a7895e580 x1874668972934144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828276.919712:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972934656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828276.919742:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972934144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828276.919750:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828276.919753:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828276.919757:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972934144:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828276.919774:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972934272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828276.919791:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828276.919802:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828276/real 1787828276] req@ffff9c1a7895fc00 x1874668972934272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828276.919806:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828276.919813:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828276.919816:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828276.919822:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4727 00000100:00100000:1.0:1787828276.919823:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972934272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828276.919826:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828276.919849:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828276.919853:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828276.919856:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972934272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828276.919860:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972934400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828276.919873:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828276.919880:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828276/real 1787828276] req@ffff9c1a7895e940 x1874668972934400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828276.919893:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972934400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828276.919898:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828276.919900:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828276.919902:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972934400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828276.919906:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972934528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828276.919917:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828276.919923:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828276/real 1787828276] req@ffff9c1a7895f480 x1874668972934528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828276.919930:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972934528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828276.919935:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828276.919937:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828276.919939:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972934528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828276.919952:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972934656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828276.919966:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828276.919973:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828276/real 1787828276] req@ffff9c1a7895cf00 x1874668972934656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828276.919982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972934656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828276.919983:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828276.919992:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828276.920002:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828276.920005:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828276.920009:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972934784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828276.920012:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972934656:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828276.920059:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828276.920062:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828276.920065:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828276.920069:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4727 00000100:00080000:3.0:1787828276.920073:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828276.920078:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972934784:192.168.203.148@tcp:8:kworker.0 00000100:00080000:3.0:1787828276.920079:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828276.920088:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828276.920101:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972934912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828276.920109:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828276.920123:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828276/real 1787828276] req@ffff9c1a7895c3c0 x1874668972934784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828276.920132:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972934784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828276.920140:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828276.920144:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828276.920146:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972934784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828276.920156:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972934912:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828276.920169:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828276.920175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828276/real 1787828276] req@ffff9c1a7895c000 x1874668972934912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828276.920183:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972934912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828276.920188:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828276.920189:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828276.920192:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972934912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828277.431410:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828277.431410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828277.687458:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828278.455461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828278.455462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828278.711488:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828279.479481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828279.479481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828279.735475:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828280.503423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828280.503423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828280.759492:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828281.527484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828281.527524:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828281.783469:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828282.039444:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828282.039452:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828282.039456:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828282.039465:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4732 00000100:00080000:3.0:1787828282.039474:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828282.039480:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828282.039514:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972935040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828282.039538:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828282.039540:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828282.039542:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828282.039545:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4732 00000100:00080000:3.0:1787828282.039547:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828282.039550:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828282.039558:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828282.039566:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972935168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828282.039575:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828282.039578:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828282.039579:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828282.039584:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4732 00000100:00100000:1.0:1787828282.039587:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972935040:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828282.039587:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828282.039593:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828282.039599:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828282.039608:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972935296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828282.039614:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828282.039617:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828282.039618:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828282.039621:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4732 00000100:00080000:3.0:1787828282.039623:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828282.039626:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828282.039629:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828282.039636:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972935424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828282.039647:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828282.039649:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828282.039651:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828282.039655:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828282.039658:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4732 00000100:00080000:3.0:1787828282.039662:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828282.039665:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828282.039669:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828282.039678:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972935552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828282.039688:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828282.039693:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828282.039694:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff9c1a7895d2c0 x1874668972935040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828282.039696:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828282.039707:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4732 00000100:00080000:3.0:1787828282.039713:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828282.039717:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972935040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828282.039722:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828282.039733:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828282.039733:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828282.039741:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828282.039744:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972935040:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828282.039749:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972935680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828282.039756:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828282.039759:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828282.039761:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828282.039766:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972935168:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828282.039766:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4732 00000100:00080000:3.0:1787828282.039773:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828282.039777:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828282.039780:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828282.039789:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972935808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828282.039793:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828282.039821:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff9c1a7895de00 x1874668972935168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.039831:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972935168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.039837:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.039843:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828282.039848:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972935168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828282.039855:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972935296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828282.039874:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828282.039881:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff9c1a7895c780 x1874668972935296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.039889:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972935296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.039894:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.039896:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828282.039898:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972935296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828282.039902:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972935424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828282.039912:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828282.039917:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff9c1a7895f840 x1874668972935424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.039925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972935424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.039930:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.039932:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828282.039934:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972935424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828282.039937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972935552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828282.039947:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828282.039953:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff9c1a7895da40 x1874668972935552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.039965:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972935552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.039970:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.039972:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828282.039974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972935552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828282.039977:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972935680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828282.039987:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828282.040001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff9c1a7895cb40 x1874668972935680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.040011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972935680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.040017:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.040019:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828282.040021:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972935680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828282.040033:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972935808:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828282.040045:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828282.040051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828282/real 1787828282] req@ffff9c1a7895f0c0 x1874668972935808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828282.040058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972935808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828282.040068:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828282.040070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828282.040072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972935808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828282.551419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828282.551426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828282.807613:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828283.575444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828283.575444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828283.831519:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828284.599418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828284.599424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828284.855492:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828285.623443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828285.623451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828285.879535:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828286.647406:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828286.647414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828286.903470:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828287.159417:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828287.159428:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828287.159433:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828287.159443:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4737 00000100:00080000:3.0:1787828287.159454:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828287.159462:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828287.159499:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972935936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828287.159529:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828287.159534:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828287.159536:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828287.159540:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4737 00000100:00080000:3.0:1787828287.159545:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828287.159550:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828287.159559:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828287.159574:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972936064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828287.159584:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828287.159590:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828287.159592:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828287.159596:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4737 00000100:00080000:3.0:1787828287.159600:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828287.159604:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828287.159610:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828287.159615:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972935936:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828287.159624:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972936192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828287.159637:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828287.159641:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828287.159643:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828287.159647:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4737 00000100:00080000:3.0:1787828287.159651:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828287.159656:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828287.159661:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828287.159673:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972936320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828287.159682:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828287.159688:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828287.159689:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828287.159694:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828287.159699:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4737 00000100:00080000:3.0:1787828287.159704:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828287.159708:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828287.159714:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828287.159728:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972936448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828287.159737:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828287.159742:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff9c1a7895e1c0 x1874668972935936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828287.159742:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828287.159747:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828287.159756:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4737 00000100:00080000:3.0:1787828287.159780:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828287.159789:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972935936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828287.159790:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828287.159799:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828287.159809:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828287.159816:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972936576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828287.159818:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828287.159828:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972935936:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828287.159837:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828287.159844:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972936064:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828287.159845:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828287.159849:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828287.159855:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4737 00000100:00080000:3.0:1787828287.159860:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828287.159864:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828287.159870:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828287.159877:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828287.159886:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972936704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828287.159893:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff9c1a7895f0c0 x1874668972936064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.159910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972936064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.159916:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.159920:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828287.159923:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972936064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828287.159943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972936192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828287.159956:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828287.159963:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff9c1a7895cb40 x1874668972936192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.159970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972936192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.159976:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.159978:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828287.159980:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972936192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828287.159983:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972936320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828287.159993:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828287.159998:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff9c1a7895da40 x1874668972936320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.160005:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972936320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.160011:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.160012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828287.160014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972936320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828287.160018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972936448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828287.160028:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828287.160033:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff9c1a7895f840 x1874668972936448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.160041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972936448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.160046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.160047:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828287.160049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972936448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828287.160052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972936576:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828287.160063:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828287.160068:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff9c1a7895c780 x1874668972936576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.160075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972936576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.160080:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.160082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828287.160083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972936576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828287.160087:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972936704:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828287.160097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828287.160103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828287/real 1787828287] req@ffff9c1a7895de00 x1874668972936704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828287.160114:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972936704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828287.160120:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828287.160121:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828287.160123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972936704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828287.671448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828287.671448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828287.927491:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828288.695427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828288.695431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828288.951507:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828289.719471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828289.719471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828289.975602:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828290.743429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828290.743430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828290.999484:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828291.767446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828291.767446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828292.023531:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828292.279481:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828292.279489:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828292.279494:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828292.279502:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4742 00000100:00080000:1.0:1787828292.279512:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828292.279518:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828292.279556:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972936832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828292.279579:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828292.279598:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828292.279599:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828292.279603:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4742 00000100:00080000:1.0:1787828292.279606:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828292.279609:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828292.279617:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828292.279627:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972936960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828292.279636:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828292.279639:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828292.279641:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828292.279643:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4742 00000100:00080000:1.0:1787828292.279645:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828292.279663:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828292.279667:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828292.279675:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972937088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828292.279687:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828292.279692:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828292.279695:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828292.279700:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4742 00000100:00080000:1.0:1787828292.279706:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828292.279711:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828292.279751:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828292.279756:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972936832:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828292.279771:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972937216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828292.279803:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828292.279807:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828292.279809:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828292.279814:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4742 00000100:00080000:1.0:1787828292.279820:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828292.279824:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828292.279829:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828292.279845:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972937344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828292.279846:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828292.279857:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828292.279860:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828292.279862:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828292.279865:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4742 00000100:00080000:1.0:1787828292.279869:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828292.279872:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828292.279875:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828292.279884:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972937472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828292.279891:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828292.279895:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828292.279897:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828292.279898:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff9c1a78a792c0 x1874668972936832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828292.279902:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4742 00000100:00080000:1.0:1787828292.279905:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828292.279911:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828292.279917:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828292.279936:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972937600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828292.279938:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972936832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828292.279984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828292.279993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828292.279999:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972936832:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828292.280008:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972936960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828292.280050:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828292.280064:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff9c1a78a7b840 x1874668972936960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828292.280077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972936960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828292.280086:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828292.280089:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828292.280093:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972936960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828292.280100:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972937088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828292.280121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828292.280132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff9c1a78a7a580 x1874668972937088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828292.280151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972937088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828292.280160:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828292.280164:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828292.280167:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972937088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828292.280195:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972937216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828292.280214:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828292.280226:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff9c1a78a78000 x1874668972937216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828292.280263:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972937216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828292.280273:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828292.280277:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828292.280280:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972937216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828292.280287:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972937344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828292.280325:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828292.280337:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff9c1a78a7b480 x1874668972937344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828292.280350:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972937344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828292.280388:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828292.280392:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828292.280396:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972937344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828292.280403:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972937472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828292.280426:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828292.280438:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff9c1a78a78f00 x1874668972937472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828292.280451:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972937472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828292.280460:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828292.280464:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828292.280467:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972937472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828292.280475:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972937600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828292.280495:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828292.280507:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828292/real 1787828292] req@ffff9c1a78a79a40 x1874668972937600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828292.280520:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972937600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828292.280529:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828292.280533:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828292.280537:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972937600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828292.791411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828292.791425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828293.047430:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828293.815419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828293.815423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828294.071483:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828294.839400:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828294.839411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828295.095470:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828295.863432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828295.863433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828296.119509:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828296.887413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828296.887421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828297.143484:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828297.399468:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828297.399478:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828297.399483:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.399493:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4747 00000100:00080000:0.0:1787828297.399506:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.399513:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828297.399553:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972937728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828297.399583:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828297.399588:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828297.399590:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.399595:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4747 00000100:00080000:0.0:1787828297.399601:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.399605:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828297.399615:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972937728:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787828297.399616:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828297.399639:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972937856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828297.399652:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828297.399655:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828297.399658:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.399662:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4747 00000100:00080000:0.0:1787828297.399667:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.399671:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828297.399678:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828297.399678:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828297.399715:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972937984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828297.399728:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828297.399728:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff9c1a78860000 x1874668972937728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828297.399738:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828297.399743:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.399750:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4747 00000100:00100000:3.0:1787828297.399754:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972937728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828297.399757:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.399767:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787828297.399769:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828297.399775:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787828297.399776:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828297.399780:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972937728:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828297.399796:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972938112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828297.399804:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972937856:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828297.399814:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828297.399819:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828297.399822:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.399827:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4747 00000800:00000100:3.0:1787828297.399829:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828297.399835:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.399842:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828297.399844:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff9c1a78863c00 x1874668972937856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828297.399850:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828297.399858:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972937856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828297.399867:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828297.399868:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972938240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828297.399874:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828297.399882:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972937856:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828297.399891:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828297.399892:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972937984:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828297.399899:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828297.399903:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828297.399908:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4747 00000100:00080000:0.0:1787828297.399913:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828297.399915:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828297.399920:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828297.399926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff9c1a788630c0 x1874668972937984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828297.399927:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828297.399940:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972937984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828297.399946:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972938368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828297.399950:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828297.399958:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828297.399964:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972937984:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828297.399965:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828297.399971:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828297.399973:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828297.399977:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972938112:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828297.399982:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4747 00000100:00080000:0.0:1787828297.399988:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828297.399993:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787828297.399998:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787828297.400001:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828297.400008:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff9c1a78861680 x1874668972938112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828297.400017:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972938496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828297.400020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972938112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828297.400052:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828297.400054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828297.400057:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972938112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828297.400064:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972938240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828297.400076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828297.400082:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff9c1a78862580 x1874668972938240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828297.400095:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972938240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828297.400100:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828297.400102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828297.400104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972938240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828297.400108:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972938368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828297.400118:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828297.400123:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff9c1a78861e00 x1874668972938368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828297.400130:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972938368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828297.400136:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828297.400137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828297.400139:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972938368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828297.400148:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972938496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828297.400157:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828297.400163:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828297/real 1787828297] req@ffff9c1a78860780 x1874668972938496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828297.400170:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972938496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828297.400179:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828297.400181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828297.400183:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972938496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828297.911444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828297.911445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828298.167473:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828298.935453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828298.935453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828299.191515:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828299.959465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828299.959465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828300.215529:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828300.983473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828300.983475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828301.239497:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828302.007450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828302.007450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828302.263569:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828302.519482:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828302.519491:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828302.519495:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828302.519503:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4752 00000100:00080000:2.0:1787828302.519513:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828302.519518:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828302.519552:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972938624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828302.519576:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828302.519582:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828302.519584:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828302.519587:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4752 00000100:00080000:2.0:1787828302.519590:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828302.519594:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828302.519603:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828302.519612:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972938752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828302.519620:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828302.519623:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828302.519627:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828302.519628:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972938624:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828302.519632:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4752 00000100:00080000:2.0:1787828302.519637:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828302.519640:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828302.519643:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828302.519652:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972938880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828302.519674:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828302.519677:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828302.519679:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828302.519682:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4752 00000100:00080000:2.0:1787828302.519697:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828302.519700:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828302.519710:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828302.519723:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972939008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828302.519731:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828302.519735:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828302.519736:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828302.519739:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4752 00000100:00080000:2.0:1787828302.519741:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828302.519746:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828302.519746:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972938752:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787828302.519754:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828302.519766:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972939136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828302.519775:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828302.519778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787828302.519779:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972938880:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828302.519780:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828302.519788:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4752 00000100:00080000:2.0:1787828302.519792:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828302.519796:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828302.519802:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828302.519810:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972939008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828302.519815:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972939264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828302.519824:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828302.519828:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828302.519829:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828302.519834:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972939136:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828302.519835:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4752 00000100:00080000:2.0:1787828302.519841:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828302.519845:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828302.519855:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828302.519859:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972939264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828302.519868:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972939392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828302.519889:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972939392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828303.031441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828303.031441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828303.287467:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828304.055411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828304.055413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828304.311496:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828305.079417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828305.079417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828305.335574:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828305.591740:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787828305.591802:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828305.591819:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828305.591857:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828305.591864:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828305.591874:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828305.591887:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828305.591900:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828305.591913:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787828305.591940:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828302/real 1787828305] req@ffff9c1a49d8c780 x1874668972939392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828305.591969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972939392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828305.591983:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828305.591989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828305.591995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972939392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828305.592006:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828302/real 1787828305] req@ffff9c1a49d8c000 x1874668972939136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828305.592029:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972939136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828305.592040:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828305.592044:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828305.592049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972939136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828305.592058:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828302/real 1787828305] req@ffff9c1a49d8de00 x1874668972939264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828305.592076:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972939264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828305.592096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828305.592101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828305.592106:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972939264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828305.592113:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828302/real 1787828305] req@ffff9c1a49d8f0c0 x1874668972938752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828305.592129:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972938752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828305.592148:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828305.592153:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828305.592158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972938752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828305.592166:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828302/real 1787828305] req@ffff9c1a49d8cf00 x1874668972938880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828305.592178:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972938880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828305.592186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828305.592190:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828305.592193:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972938880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828305.592200:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828302/real 1787828305] req@ffff9c1a49d8da40 x1874668972939008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828305.592211:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972939008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828305.592219:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828305.592222:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828305.592225:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972939008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828305.592232:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828302/real 1787828305] req@ffff9c1a49d8cb40 x1874668972938624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828305.592243:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972938624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828305.592250:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828305.592254:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828305.592257:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972938624:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828306.103427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828306.103428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828306.359541:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828307.127411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828307.127425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828307.383468:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828307.639404:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828307.639414:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828307.639418:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828307.639429:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4757 00000100:00080000:2.0:1787828307.639440:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828307.639449:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828307.639487:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972939520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828307.639523:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828307.639527:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828307.639530:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828307.639534:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4757 00000100:00080000:2.0:1787828307.639538:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828307.639542:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828307.639552:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828307.639566:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972939648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828307.639569:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972939520:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828307.639586:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828307.639590:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828307.639592:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828307.639597:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4757 00000100:00080000:2.0:1787828307.639600:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828307.639604:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828307.639610:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828307.639640:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972939776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828307.639652:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828307.639658:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828307.639659:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828307.639663:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828307.639669:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4757 00000100:00080000:2.0:1787828307.639675:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828307.639680:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828307.639685:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828307.639700:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972939904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828307.639709:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828307.639713:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828307.639715:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828307.639721:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4757 00000100:00100000:1.0:1787828307.639722:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828307/real 1787828307] req@ffff9c1a49d8cb40 x1874668972939520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828307.639725:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828307.639733:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828307.639742:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828307.639756:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972939520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828307.639763:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972940032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828307.639777:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828307.639781:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828307.639786:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787828307.639788:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828307.639792:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828307.639795:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972939520:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828307.639800:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4757 00000100:00080000:2.0:1787828307.639807:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828307.639812:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828307.639817:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828307.639829:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972940160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828307.639844:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972939648:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828307.639851:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828307.639857:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828307.639859:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828307.639865:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4757 00000100:00080000:2.0:1787828307.639870:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828307.639875:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787828307.639882:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787828307.639882:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828307.639899:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972940288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828307.639904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828307/real 1787828307] req@ffff9c1a49d8da40 x1874668972939648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828307.639928:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972939648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828307.639937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828307.639941:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828307.639945:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972939648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828307.639955:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972939776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828307.639979:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828307.639991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828307/real 1787828307] req@ffff9c1a49d8cf00 x1874668972939776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828307.640003:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972939776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828307.640011:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828307.640014:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828307.640016:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972939776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828307.640022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972939904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828307.640041:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828307.640051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828307/real 1787828307] req@ffff9c1a49d8f0c0 x1874668972939904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828307.640070:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972939904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828307.640079:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828307.640082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828307.640085:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972939904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828307.640092:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972940032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828307.640113:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828307.640125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828307/real 1787828307] req@ffff9c1a49d8de00 x1874668972940032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828307.640138:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972940032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828307.640147:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828307.640150:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828307.640154:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972940032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828307.640175:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972940160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828307.640216:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828307.640228:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828307/real 1787828307] req@ffff9c1a49d8c000 x1874668972940160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828307.640241:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972940160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828307.640256:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828307.640260:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828307.640263:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972940160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828307.640270:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972940288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828307.640290:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828307.640301:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828307/real 1787828307] req@ffff9c1a49d8c780 x1874668972940288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828307.640315:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972940288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828307.640324:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828307.640328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828307.640331:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972940288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828308.151417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828308.151420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828308.407578:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828309.175448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828309.175464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828309.431515:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828310.199442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828310.199457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828310.455504:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828311.223461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828311.223461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828311.479571:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828312.247424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828312.247444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828312.503626:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828312.759462:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828312.759471:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828312.759476:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828312.759485:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4763 00000100:00080000:0.0:1787828312.759495:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828312.759502:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828312.759532:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972940416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828312.759557:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828312.759560:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828312.759562:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828312.759566:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4763 00000100:00080000:0.0:1787828312.759569:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828312.759573:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828312.759583:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828312.759601:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972940416:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828312.759614:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972940544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828312.759634:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828312.759638:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828312.759640:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828312.759645:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4763 00000100:00080000:0.0:1787828312.759649:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828312.759653:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828312.759662:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828312.759668:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828312.759676:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972940672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828312.759689:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828312.759693:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828312.759699:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828312.759706:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828312/real 1787828312] req@ffff9c1a788621c0 x1874668972940416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828312.759707:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4763 00000100:00080000:0.0:1787828312.759713:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828312.759722:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828312.759727:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828312.759729:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972940416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828312.759740:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828312.759744:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972940800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828312.759746:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828312.759755:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972940416:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828312.759771:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828312.759775:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828312.759777:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828312.759781:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4763 00000100:00080000:0.0:1787828312.759785:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828312.759787:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972940544:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828312.759792:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828312.759802:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828312.759818:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972940928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828312.759828:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828312.759829:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828312.759835:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828312.759838:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828312.759842:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4763 00000100:00080000:0.0:1787828312.759847:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828312.759850:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828312/real 1787828312] req@ffff9c1a78862940 x1874668972940544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828312.759854:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828312.759870:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828312.759875:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972940544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828312.759884:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828312.759888:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972941056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828312.759889:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828312.759896:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972940544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828312.759906:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972940672:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828312.759912:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828312.759916:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828312.759919:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828312.759924:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4763 00000800:00000100:2.0:1787828312.759929:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828312.759931:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828312.759941:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828312.759943:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828312/real 1787828312] req@ffff9c1a78861a40 x1874668972940672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828312.759949:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828312.759955:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972940672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828312.759963:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828312.759967:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972941184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828312.759968:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828312.759974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972940672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828312.759985:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972940800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828312.760010:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828312.760025:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828312/real 1787828312] req@ffff9c1a78860f00 x1874668972940800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828312.760039:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972940800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828312.760049:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828312.760052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828312.760056:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972940800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828312.760074:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972940928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828312.760096:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828312.760108:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828312/real 1787828312] req@ffff9c1a78862d00 x1874668972940928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828312.760121:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972940928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828312.760127:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828312.760129:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828312.760131:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972940928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828312.760137:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972941056:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828312.760159:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828312.760171:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828312/real 1787828312] req@ffff9c1a78860b40 x1874668972941056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828312.760184:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972941056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828312.760195:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828312.760197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828312.760200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972941056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828312.760204:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972941184:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828312.760220:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828312.760227:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828312/real 1787828312] req@ffff9c1a78863840 x1874668972941184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828312.760235:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972941184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828312.760241:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828312.760243:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828312.760245:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972941184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828313.271456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828313.271462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828313.527555:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828314.295418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828314.295423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828314.551475:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828315.319431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828315.319439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828315.575516:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828316.343473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828316.343486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828316.599506:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828317.367467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828317.367537:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828317.623516:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828317.879483:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828317.879495:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828317.879500:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828317.879511:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4768 00000100:00080000:3.0:1787828317.879523:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828317.879530:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828317.879626:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972941312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828317.879660:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828317.879665:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828317.879667:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828317.879672:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4768 00000100:00080000:3.0:1787828317.879704:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828317.879711:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828317.879737:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828317.879775:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972941440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828317.879787:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972941312:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828317.879801:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828317.879833:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828317.879836:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828317.879843:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4768 00000100:00080000:3.0:1787828317.879849:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828317.879854:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828317.879861:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828317.879875:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972941568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828317.879894:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828317.879897:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828317.879901:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828317.879904:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828317.879909:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4768 00000100:00080000:3.0:1787828317.879915:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828317.879919:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828317.879923:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828317.879935:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972941696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828317.879941:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828317/real 1787828317] req@ffff9c1a7895d2c0 x1874668972941312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828317.879945:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828317.879953:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828317.879958:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828317.879964:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4768 00000100:00080000:3.0:1787828317.879969:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828317.879974:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828317.879982:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828317.879992:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972941824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828317.880002:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972941312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828317.880002:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828317.880009:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828317.880012:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828317.880019:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4768 00000100:00080000:3.0:1787828317.880024:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828317.880037:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828317.880042:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828317.880042:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828317.880053:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972941952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828317.880054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828317.880083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972941312:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828317.880085:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828317.880095:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828317.880099:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828317.880105:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4768 00000100:00080000:3.0:1787828317.880109:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828317.880112:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828317.880117:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828317.880137:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972942080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828317.880145:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972941440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828317.880182:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828317.880200:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828317/real 1787828317] req@ffff9c1a7895c000 x1874668972941440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828317.880213:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972941440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828317.880221:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828317.880225:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828317.880228:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972941440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828317.880234:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972941568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828317.880250:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828317.880259:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828317/real 1787828317] req@ffff9c1a7895c3c0 x1874668972941568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828317.880270:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972941568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828317.880284:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828317.880287:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828317.880290:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972941568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828317.880295:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972941696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828317.880309:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828317.880317:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828317/real 1787828317] req@ffff9c1a7895cf00 x1874668972941696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828317.880327:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972941696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828317.880335:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828317.880337:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828317.880340:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972941696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828317.880345:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972941824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828317.880359:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828317.880367:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828317/real 1787828317] req@ffff9c1a7895f480 x1874668972941824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828317.880410:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972941824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828317.880418:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828317.880438:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828317.880443:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972941824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828317.880450:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972941952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828317.880474:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828317.880485:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828317/real 1787828317] req@ffff9c1a7895e940 x1874668972941952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828317.880497:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972941952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828317.880506:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828317.880509:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828317.880512:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972941952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828317.880532:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972942080:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828317.880588:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828317.880598:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828317/real 1787828317] req@ffff9c1a7895fc00 x1874668972942080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828317.880617:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972942080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828317.880623:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828317.880626:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828317.880628:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972942080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828318.391621:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828318.391638:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828318.647511:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828319.415524:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828319.415562:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828319.671501:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828320.439432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828320.439478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828320.695464:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828321.463461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828321.463471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828321.719430:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828322.487498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828322.487506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828322.743624:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828322.999641:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828322.999652:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828322.999658:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828322.999669:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4773 00000100:00080000:0.0:1787828322.999685:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828322.999697:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828322.999777:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972942208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828322.999817:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828322.999821:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828322.999823:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828322.999828:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4773 00000100:00080000:0.0:1787828322.999832:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828322.999837:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828322.999847:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828322.999866:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972942336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828322.999883:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828322.999888:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828322.999891:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828322.999902:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4773 00000100:00100000:2.0:1787828322.999903:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972942208:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828322.999912:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828322.999922:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828322.999931:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828322.999952:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972942464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828322.999966:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828322.999970:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828322.999972:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828322.999975:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4773 00000100:00080000:0.0:1787828322.999980:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828322.999983:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828322.999989:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828322.999996:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828323.000009:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972942592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828323.000019:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828323.000023:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828323.000025:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828323.000029:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4773 00000100:00100000:2.0:1787828323.000034:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828322/real 1787828322] req@ffff9c1a788612c0 x1874668972942208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828323.000035:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828323.000044:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828323.000053:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828323.000069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972942208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828323.000075:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972942720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828323.000093:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828323.000121:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828323.000124:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828323.000132:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972942208:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828323.000141:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828323.000149:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828323.000150:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972942336:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828323.000170:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4773 00000100:00080000:0.0:1787828323.000181:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828323.000189:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828323.000199:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787828323.000200:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828323.000221:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828322/real 1787828323] req@ffff9c1a78863840 x1874668972942336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828323.000229:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972942848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828323.000257:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972942336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828323.000258:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828323.000275:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828323.000281:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828323.000295:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4773 00000100:00080000:2.0:1787828323.000296:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828323.000321:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828323.000324:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828323.000330:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828323.000334:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972942336:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787828323.000338:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828323.000353:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972942976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828323.000371:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972942464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828323.000483:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828323.000499:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828323/real 1787828323] req@ffff9c1a78860b40 x1874668972942464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828323.000552:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972942464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828323.000560:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828323.000563:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828323.000566:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972942464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828323.000572:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972942592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828323.000611:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828323.000624:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828323/real 1787828323] req@ffff9c1a78862d00 x1874668972942592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828323.000637:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972942592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828323.000646:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828323.000649:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828323.000652:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972942592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828323.000659:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972942720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828323.000676:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828323.000686:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828323/real 1787828323] req@ffff9c1a78860f00 x1874668972942720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828323.000707:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972942720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828323.000732:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828323.000736:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828323.000740:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972942720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828323.000747:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972942848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828323.000769:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828323.000780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828323/real 1787828323] req@ffff9c1a78861a40 x1874668972942848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828323.000807:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972942848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828323.000815:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828323.000817:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828323.000820:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972942848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828323.000825:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972942976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828323.000840:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828323.000848:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828323/real 1787828323] req@ffff9c1a78862940 x1874668972942976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828323.000858:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972942976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828323.000872:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828323.000875:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828323.000878:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972942976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828323.511427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828323.511462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828323.767575:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828324.535489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828324.535562:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828324.791618:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828325.559442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828325.559477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828325.815483:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828326.583447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828326.583447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828326.839536:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828327.607423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828327.607513:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828327.863454:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828328.119446:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828328.119454:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828328.119458:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828328.119467:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4778 00000100:00080000:3.0:1787828328.119489:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828328.119497:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828328.119544:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972943104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828328.119566:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828328.119568:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828328.119570:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828328.119573:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4778 00000100:00080000:3.0:1787828328.119576:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828328.119589:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828328.119597:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828328.119607:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972943232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828328.119614:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828328.119617:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828328.119618:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828328.119620:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4778 00000100:00080000:3.0:1787828328.119622:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828328.119624:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828328.119627:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828328.119634:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972943104:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828328.119636:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972943360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828328.119648:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828328.119651:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828328.119652:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828328.119654:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4778 00000100:00080000:3.0:1787828328.119657:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828328.119660:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828328.119663:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828328.119671:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972943488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828328.119678:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828328.119680:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828328.119682:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828328.119684:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4778 00000100:00080000:3.0:1787828328.119696:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828328.119698:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828328.119701:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828328.119709:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828328.119710:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972943616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828328.119718:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828328.119721:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828328.119722:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828328.119725:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4778 00000100:00080000:3.0:1787828328.119728:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828328.119730:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828328.119734:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828328.119743:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972943744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828328.119753:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828328.119757:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828328.119759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828328.119763:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4778 00000100:00100000:2.0:1787828328.119767:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff9c1a7895e580 x1874668972943104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828328.119770:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828328.119780:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828328.119790:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828328.119812:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972943104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828328.119815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972943872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828328.119845:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828328.119851:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828328.119855:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972943104:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828328.119862:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972943232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828328.119883:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828328.119893:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff9c1a7895fc00 x1874668972943232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828328.119902:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972943232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828328.119916:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828328.119919:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828328.119921:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972943232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828328.119941:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972943360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828328.119954:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828328.119961:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff9c1a7895e940 x1874668972943360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828328.119968:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972943360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828328.119974:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828328.119976:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828328.119978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972943360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828328.119982:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972943488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828328.119992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828328.119997:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff9c1a7895f480 x1874668972943488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828328.120005:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972943488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828328.120010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828328.120012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828328.120014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972943488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828328.120018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972943616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828328.120029:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828328.120034:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff9c1a7895cf00 x1874668972943616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828328.120041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972943616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828328.120047:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828328.120048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828328.120050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972943616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828328.120054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972943744:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828328.120065:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828328.120071:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff9c1a7895c3c0 x1874668972943744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828328.120078:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972943744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828328.120083:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828328.120085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828328.120087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972943744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828328.120102:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972943872:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828328.120112:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828328.120118:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828328/real 1787828328] req@ffff9c1a7895c000 x1874668972943872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828328.120126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972943872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828328.120131:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828328.120133:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828328.120135:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972943872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828328.631428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828328.631519:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828328.887539:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828329.655458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828329.655458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828329.911547:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828330.679544:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828330.679576:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828330.935704:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828331.703442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828331.703476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828331.959479:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828332.727410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828332.727414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828332.983589:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828333.239567:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828333.239581:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828333.239586:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828333.239610:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4783 00000100:00080000:3.0:1787828333.239625:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828333.239633:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828333.239684:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972944000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828333.239738:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828333.239744:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828333.239747:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828333.239753:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4783 00000100:00080000:3.0:1787828333.239758:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828333.239764:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828333.239778:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828333.239800:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972944128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828333.239813:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972944000:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828333.239838:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828333.239843:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828333.239847:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828333.239854:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4783 00000100:00080000:3.0:1787828333.239861:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828333.239867:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828333.239875:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828333.239898:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972944256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828333.239912:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828333.239919:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828333.239925:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828333.239927:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828333.239933:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4783 00000100:00080000:3.0:1787828333.239939:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828333.239944:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828333.239950:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828333.239964:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972944384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828333.239974:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828333.239979:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828333.239982:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828333.239984:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff9c1a7895d2c0 x1874668972944000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828333.239989:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4783 00000100:00080000:3.0:1787828333.239996:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828333.240003:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828333.240013:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828333.240028:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972944512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828333.240038:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828333.240041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972944000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828333.240042:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828333.240047:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828333.240076:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4783 00000100:00080000:3.0:1787828333.240087:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828333.240098:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828333.240104:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828333.240115:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972944640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828333.240125:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828333.240131:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828333.240137:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828333.240143:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828333.240144:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828333.240158:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4783 00000100:00100000:2.0:1787828333.240165:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972944000:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828333.240166:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828333.240178:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828333.240187:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828333.240200:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972944768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828333.240230:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972944128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828333.240298:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828333.240329:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff9c1a7895de00 x1874668972944128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828333.240416:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972944128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828333.240436:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828333.240471:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828333.240480:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972944128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828333.240496:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972944256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828333.240540:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828333.240567:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff9c1a7895c780 x1874668972944256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828333.240594:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972944256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828333.240612:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828333.240619:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828333.240626:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972944256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828333.240640:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972944384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828333.240682:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828333.240722:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff9c1a7895f840 x1874668972944384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828333.240753:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972944384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828333.240767:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828333.240772:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828333.240778:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972944384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828333.240787:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972944512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828333.240817:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828333.240834:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff9c1a7895da40 x1874668972944512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828333.240853:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972944512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828333.240888:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828333.240893:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828333.240898:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972944512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828333.240908:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972944640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828333.240940:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828333.240958:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff9c1a7895cb40 x1874668972944640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828333.240977:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972944640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828333.241007:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828333.241012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828333.241018:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972944640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828333.241050:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972944768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828333.241100:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828333.241120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828333/real 1787828333] req@ffff9c1a7895f0c0 x1874668972944768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828333.241139:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972944768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828333.241153:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828333.241157:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828333.241162:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972944768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828333.751418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828333.751418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828334.007494:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828334.775456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828334.775474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828335.031513:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828335.799535:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828335.799572:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828336.055533:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828336.823540:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828336.823540:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828337.079737:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828337.847577:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828337.847594:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828338.103506:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828338.359457:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828338.359471:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828338.359477:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828338.359491:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4788 00000100:00080000:0.0:1787828338.359505:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828338.359515:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828338.359566:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972944896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828338.359597:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828338.359601:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828338.359603:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828338.359608:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4788 00000100:00080000:0.0:1787828338.359612:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828338.359616:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828338.359629:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828338.359646:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972945024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828338.359647:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972944896:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828338.359673:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828338.359677:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828338.359679:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828338.359684:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4788 00000100:00080000:0.0:1787828338.359688:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828338.359693:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828338.359699:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828338.359712:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972945152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828338.359724:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828338.359730:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828338.359731:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828338.359734:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828338.359740:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4788 00000100:00080000:0.0:1787828338.359745:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828338.359750:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828338.359756:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828338.359767:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972945280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828338.359778:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828338.359784:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828338.359784:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff9c1a788621c0 x1874668972944896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828338.359786:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828338.359794:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4788 00000100:00080000:0.0:1787828338.359801:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828338.359808:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828338.359814:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828338.359820:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972944896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828338.359829:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972945408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828338.359850:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828338.359853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828338.359857:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828338.359858:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828338.359863:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828338.359864:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972944896:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828338.359870:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4788 00000100:00080000:0.0:1787828338.359874:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828338.359879:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828338.359884:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828338.359894:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972945024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828338.359897:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972945536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828338.359911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828338.359915:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828338.359917:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828338.359920:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828338.359925:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4788 00000100:00080000:0.0:1787828338.359930:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828338.359935:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff9c1a78860780 x1874668972945024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828338.359937:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828338.359946:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828338.359951:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972945024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828338.359959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828338.359961:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972945664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828338.359964:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828338.359970:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972945024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828338.359981:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972945152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828338.360000:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828338.360009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff9c1a78861e00 x1874668972945152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828338.360017:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972945152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828338.360024:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828338.360026:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828338.360028:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972945152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828338.360033:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972945280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828338.360043:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828338.360049:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff9c1a78862580 x1874668972945280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828338.360057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972945280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828338.360063:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828338.360064:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828338.360066:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972945280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828338.360069:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972945408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828338.360079:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828338.360085:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff9c1a78861680 x1874668972945408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828338.360092:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972945408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828338.360097:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828338.360099:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828338.360100:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972945408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828338.360113:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972945536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828338.360124:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828338.360130:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff9c1a788630c0 x1874668972945536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828338.360139:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972945536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828338.360144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828338.360146:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828338.360148:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972945536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828338.360151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972945664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828338.360161:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828338.360166:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828338/real 1787828338] req@ffff9c1a78863c00 x1874668972945664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828338.360174:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972945664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828338.360179:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828338.360181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828338.360182:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972945664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828338.871431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828338.871431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828339.127534:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828339.895536:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828339.895536:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828340.151598:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828340.919507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828340.919524:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828341.175603:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828341.943520:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828341.943551:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828342.199603:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828342.967460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828342.967468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828343.223574:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828343.479559:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828343.479573:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828343.479580:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828343.479595:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4793 00000100:00080000:2.0:1787828343.479612:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828343.479623:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828343.479692:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972945792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828343.479726:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828343.479730:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828343.479731:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828343.479735:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4793 00000100:00080000:2.0:1787828343.479738:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828343.479742:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828343.479753:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828343.479763:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972945920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828343.479773:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828343.479776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828343.479778:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828343.479779:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972945792:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828343.479783:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4793 00000100:00080000:2.0:1787828343.479788:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828343.479793:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828343.479796:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828343.479804:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972946048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828343.479812:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828343.479815:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828343.479817:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828343.479820:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4793 00000100:00080000:2.0:1787828343.479822:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828343.479825:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828343.479833:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828343.479842:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972946176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828343.479848:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828343.479850:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828343.479852:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828343.479855:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4793 00000100:00080000:2.0:1787828343.479858:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828343.479860:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828343.479863:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828343.479869:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828343.479873:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972946304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828343.479881:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828343.479883:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828343.479885:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828343.479888:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4793 00000100:00080000:2.0:1787828343.479891:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828343.479894:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828343.479897:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828343.479906:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972946432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828343.479911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828343.479914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828343.479916:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828343.479921:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4793 00000100:00100000:1.0:1787828343.479926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828343/real 1787828343] req@ffff9c1a49d8e1c0 x1874668972945792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828343.479927:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828343.479937:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828343.479964:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828343.479972:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972945792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828343.479987:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828343.479995:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972946560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828343.479996:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828343.480017:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972945792:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828343.480057:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972945920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828343.480087:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828343.480105:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828343/real 1787828343] req@ffff9c1a49d8c3c0 x1874668972945920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828343.480128:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972945920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828343.480155:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828343.480159:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828343.480163:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972945920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828343.480169:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972946048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828343.480190:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828343.480217:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828343/real 1787828343] req@ffff9c1a49d8e580 x1874668972946048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828343.480241:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972946048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828343.480251:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828343.480254:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828343.480258:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972946048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828343.480265:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972946176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828343.480287:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828343.480299:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828343/real 1787828343] req@ffff9c1a49d8fc00 x1874668972946176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828343.480312:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972946176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828343.480322:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828343.480326:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828343.480330:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972946176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828343.480336:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972946304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828343.480353:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828343.480364:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828343/real 1787828343] req@ffff9c1a49d8ed00 x1874668972946304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828343.480479:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972946304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828343.480496:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828343.480500:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828343.480503:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972946304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828343.480510:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972946432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828343.480536:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828343.480549:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828343/real 1787828343] req@ffff9c1a49d8d680 x1874668972946432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828343.480561:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972946432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828343.480569:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828343.480572:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828343.480574:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972946432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828343.480580:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972946560:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828343.480596:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828343.480605:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828343/real 1787828343] req@ffff9c1a49d8c780 x1874668972946560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828343.480615:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972946560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828343.480623:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828343.480626:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828343.480630:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972946560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828343.991557:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828343.991557:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828344.247595:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828345.015471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828345.015509:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828345.271640:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828346.039541:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828346.039624:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828346.295658:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828347.063486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828347.063503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828347.319515:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828348.087506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828348.087518:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828348.343595:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828348.599558:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828348.599574:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828348.599581:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828348.599597:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4798 00000100:00080000:2.0:1787828348.599616:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828348.599627:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828348.599705:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972946688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828348.599787:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828348.599795:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828348.599798:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828348.599805:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4798 00000100:00080000:2.0:1787828348.599811:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828348.599817:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828348.599831:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828348.599856:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972946816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828348.599890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828348.599898:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972946688:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828348.599898:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828348.599909:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828348.599927:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4798 00000100:00080000:2.0:1787828348.599932:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828348.599938:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828348.599945:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828348.599972:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972946944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828348.600004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828348.600013:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787828348.600014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828348.600023:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828348.600037:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4798 00000100:00080000:2.0:1787828348.600050:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828348.600057:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828348.600066:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828348.600074:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828348/real 1787828348] req@ffff9c1a49d8c000 x1874668972946688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828348.600089:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972947072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828348.600103:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972946688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828348.600108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828348.600132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828348.600139:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828348.600158:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828348.600167:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4798 00000100:00080000:1.0:1787828348.600173:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828348.600184:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972946688:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828348.600184:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828348.600210:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828348.600222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828348.600237:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972946816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828348.600246:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972947200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828348.600269:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828348.600274:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828348.600278:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828348.600283:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4798 00000800:00000100:1.0:1787828348.600297:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828348.600303:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828348.600313:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828348.600323:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828348/real 1787828348] req@ffff9c1a49d8de00 x1874668972946816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828348.600323:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828348.600471:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972946816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828348.600486:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828348.600492:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828348.600501:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972946816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828348.600513:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972947328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828348.600521:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972946944:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828348.600563:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828348.600568:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828348.600573:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828348.600583:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4798 00000100:00080000:2.0:1787828348.600593:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828348.600595:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828348.600611:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828348.600622:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828348.600626:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828348/real 1787828348] req@ffff9c1a49d8f0c0 x1874668972946944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828348.600673:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972947456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828348.600677:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972946944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828348.600751:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828348.600758:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828348.600764:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972946944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828348.600773:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972947072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828348.600798:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828348.600811:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828348/real 1787828348] req@ffff9c1a49d8cf00 x1874668972947072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828348.600822:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972947072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828348.600827:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828348.600829:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828348.600831:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972947072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828348.600849:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972947200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828348.600863:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828348.600878:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828348/real 1787828348] req@ffff9c1a49d8da40 x1874668972947200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828348.600904:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972947200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828348.600909:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828348.600912:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828348.600914:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972947200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828348.600919:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972947328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828348.600940:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828348.600948:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828348/real 1787828348] req@ffff9c1a49d8cb40 x1874668972947328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828348.600955:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972947328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828348.600961:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828348.600963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828348.600965:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972947328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828348.600970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972947456:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828348.600987:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828348.601016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828348/real 1787828348] req@ffff9c1a49d8c000 x1874668972947456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828348.601025:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972947456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828348.601038:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828348.601042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828348.601048:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972947456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828349.111440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828349.111449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828349.367552:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828350.135435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828350.135441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828350.391542:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828351.159422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828351.159432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828351.415492:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828352.183501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828352.183544:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828352.439667:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828353.207467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828353.207481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828353.463617:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828353.719496:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828353.719515:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828353.719523:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828353.719541:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4804 00000100:00080000:2.0:1787828353.719561:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828353.719574:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828353.719642:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972947584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828353.719694:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828353.719702:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828353.719706:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828353.719716:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4804 00000100:00080000:2.0:1787828353.719725:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828353.719738:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828353.719757:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828353.719769:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972947584:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828353.719786:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972947712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828353.719803:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828353.719808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828353.719811:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828353.719816:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4804 00000100:00080000:2.0:1787828353.719822:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828353.719827:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828353.719833:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828353.719848:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972947840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828353.719864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828353.719869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828353.719871:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828353.719879:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4804 00000800:00000100:1.0:1787828353.719887:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828353.719887:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828353.719904:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828353.719911:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828353.719925:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972947968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828353.719937:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828353.719943:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828353.719947:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828353.719956:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828353/real 1787828353] req@ffff9c1a49d8c780 x1874668972947584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828353.719961:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4804 00000100:00080000:2.0:1787828353.719975:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828353.719991:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828353.720000:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828353.720010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972947584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828353.720018:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972948096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828353.720050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828353.720058:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828353.720065:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828353.720081:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4804 00000100:00080000:2.0:1787828353.720089:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828353.720090:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828353.720101:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828353.720102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:2.0:1787828353.720120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828353.720123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972947584:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828353.720140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972948224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828353.720152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828353.720157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828353.720160:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828353.720166:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972947712:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828353.720173:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4804 00000100:00080000:2.0:1787828353.720186:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828353.720193:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828353.720202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828353.720216:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828353.720222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972948352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828353.720235:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828353/real 1787828353] req@ffff9c1a49d8d680 x1874668972947712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828353.720260:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972947712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.720270:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.720274:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828353.720279:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972947712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828353.720285:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972947840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828353.720321:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828353.720332:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828353/real 1787828353] req@ffff9c1a49d8ed00 x1874668972947840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828353.720342:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972947840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.720380:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.720384:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828353.720388:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972947840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828353.720398:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972947968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828353.720426:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828353.720441:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828353/real 1787828353] req@ffff9c1a49d8fc00 x1874668972947968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828353.720456:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972947968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.720466:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.720470:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828353.720473:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972947968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828353.720481:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972948096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828353.720505:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828353.720521:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828353/real 1787828353] req@ffff9c1a49d8e580 x1874668972948096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828353.720537:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972948096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.720549:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.720554:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828353.720557:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972948096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828353.720564:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972948224:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828353.720585:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828353.720594:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828353/real 1787828353] req@ffff9c1a49d8c3c0 x1874668972948224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828353.720604:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972948224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.720611:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.720613:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828353.720616:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972948224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828353.720635:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972948352:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828353.720648:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828353.720655:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828353/real 1787828353] req@ffff9c1a49d8e1c0 x1874668972948352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828353.720664:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972948352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828353.720672:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828353.720674:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828353.720677:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972948352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828354.231438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828354.231494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828354.487599:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828355.255437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828355.255496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828355.511679:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828356.279465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828356.279517:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828356.535653:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828357.303508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828357.303561:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828357.559471:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828358.327433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828358.327487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828358.583648:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828358.839601:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828358.839619:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828358.839626:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828358.839639:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4809 00000100:00080000:3.0:1787828358.839654:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828358.839664:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828358.839715:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972948480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828358.839757:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828358.839763:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828358.839765:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828358.839770:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4809 00000100:00080000:3.0:1787828358.839774:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828358.839779:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828358.839793:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828358.839811:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972948608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828358.839827:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828358.839831:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828358.839834:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828358.839843:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4809 00000100:00100000:1.0:1787828358.839847:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972948480:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828358.839848:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828358.839857:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828358.839869:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828358.839885:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972948736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828358.839899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828358.839903:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828358.839906:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828358.839910:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4809 00000100:00080000:3.0:1787828358.839914:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828358.839918:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828358.839925:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828358.839939:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972948864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828358.839950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828358.839957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1787828358.839962:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828358.839963:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828358.839976:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4809 00000100:00080000:3.0:1787828358.839983:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828358.839989:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828358.839996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828358.840011:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972948992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828358.840023:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828358.840027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828358.840030:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828358.840038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828358/real 1787828358] req@ffff9c1a7895e1c0 x1874668972948480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828358.840042:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4809 00000100:00080000:3.0:1787828358.840053:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828358.840065:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828358.840076:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828358.840095:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972949120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828358.840106:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972948480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828358.840112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828358.840122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828358.840127:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828358.840141:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4809 00000100:00080000:3.0:1787828358.840149:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828358.840162:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828358.840170:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828358.840174:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828358.840186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972949248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828358.840188:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828358.840204:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972948480:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828358.840252:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972948608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828358.840324:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828358.840407:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828358/real 1787828358] req@ffff9c1a7895f0c0 x1874668972948608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828358.840427:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972948608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828358.840439:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828358.840444:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828358.840448:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972948608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828358.840457:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972948736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828358.840484:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828358.840498:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828358/real 1787828358] req@ffff9c1a7895cb40 x1874668972948736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828358.840513:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972948736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828358.840523:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828358.840527:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828358.840531:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972948736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828358.840538:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972948864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828358.840557:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828358.840568:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828358/real 1787828358] req@ffff9c1a7895da40 x1874668972948864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828358.840586:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972948864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828358.840604:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828358.840609:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828358.840615:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972948864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828358.840627:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972948992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828358.840666:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828358.840698:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828358/real 1787828358] req@ffff9c1a7895f840 x1874668972948992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828358.840715:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972948992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828358.840725:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828358.840729:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828358.840733:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972948992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828358.840741:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972949120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828358.840768:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828358.840780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828358/real 1787828358] req@ffff9c1a7895c780 x1874668972949120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828358.840794:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972949120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828358.840804:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828358.840807:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828358.840811:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972949120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828358.840822:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972949248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828358.840855:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828358.840876:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828358/real 1787828358] req@ffff9c1a7895de00 x1874668972949248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828358.840917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972949248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828358.840929:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828358.840933:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828358.840937:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972949248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828359.351598:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828359.351629:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828359.607465:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828360.375417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828360.375423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828360.631588:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828361.399485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828361.399541:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828361.655521:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828362.423467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828362.423495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828362.679609:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828363.447515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828363.447546:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828363.703617:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828363.959515:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828363.959528:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828363.959534:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828363.959547:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4814 00000100:00080000:2.0:1787828363.959561:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828363.959570:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828363.959623:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668972949376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828363.959664:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828363.959669:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828363.959672:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828363.959679:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4814 00000100:00080000:2.0:1787828363.959683:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828363.959688:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828363.959700:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828363.959715:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668972949504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828363.959731:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828363.959735:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828363.959737:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828363.959742:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4814 00000100:00080000:2.0:1787828363.959746:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828363.959751:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828363.959760:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828363.959775:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668972949632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828363.959781:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972949376:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828363.959803:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828363.959808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828363.959810:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828363.959816:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4814 00000100:00080000:2.0:1787828363.959820:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828363.959825:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828363.959832:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828363.959854:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668972949760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828363.959875:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828363.959882:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828363.959886:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828363.959900:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828363.959910:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4814 00000100:00080000:2.0:1787828363.959922:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828363.959933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828363.959946:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828363.959960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828363/real 1787828363] req@ffff9c1a49d8e1c0 x1874668972949376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828363.959969:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668972949888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828363.960028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828363.960032:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828363.960034:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828363.960037:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668972949376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828363.960046:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4814 00000100:00080000:2.0:1787828363.960065:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828363.960071:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828363.960075:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828363.960084:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:2.0:1787828363.960088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828363.960096:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972949376:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828363.960111:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668972950016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828363.960113:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972949504:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828363.960147:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828363.960151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828363.960155:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828363.960164:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828363.960164:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4814 00000100:00080000:2.0:1787828363.960180:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828363.960188:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828363.960193:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828363/real 1787828363] req@ffff9c1a49d8c3c0 x1874668972949504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828363.960201:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828363.960223:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668972949504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828363.960256:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972950144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828363.960258:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828363.960278:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828363.960295:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972949504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828363.960406:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972949632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828363.960429:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828363.960437:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828363/real 1787828363] req@ffff9c1a49d8e580 x1874668972949632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828363.960446:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668972949632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828363.960452:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828363.960455:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828363.960457:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972949632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828363.960461:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972949760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828363.960473:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828363.960480:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828363/real 1787828363] req@ffff9c1a49d8fc00 x1874668972949760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828363.960487:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668972949760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828363.960493:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828363.960495:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828363.960498:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972949760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828363.960503:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972949888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828363.960513:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828363.960519:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828363/real 1787828363] req@ffff9c1a49d8ed00 x1874668972949888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828363.960526:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668972949888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828363.960532:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828363.960534:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828363.960536:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972949888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828363.960540:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972950016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828363.960551:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828363.960557:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828363/real 1787828363] req@ffff9c1a49d8d680 x1874668972950016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828363.960565:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668972950016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828363.960570:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828363.960572:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828363.960574:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972950016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828363.960578:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972950144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828363.960588:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828363.960595:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828363/real 1787828363] req@ffff9c1a49d8c780 x1874668972950144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828363.960602:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972950144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828363.960608:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828363.960610:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828363.960612:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972950144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828364.471450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828364.471453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828364.727554:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828365.495473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828365.495473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828365.751559:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828366.519449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828366.519463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828366.775563:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828367.543418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828367.543483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828367.799619:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828368.567471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828368.567477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828368.823640:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828369.079509:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828369.079525:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828369.079533:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828369.079550:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4819 00000100:00080000:3.0:1787828369.079568:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828369.079579:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828369.079657:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972950272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828369.079705:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828369.079711:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828369.079713:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828369.079719:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4819 00000100:00080000:3.0:1787828369.079724:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828369.079730:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828369.079745:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828369.079762:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972950400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828369.079787:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828369.079794:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828369.079797:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828369.079802:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4819 00000100:00080000:3.0:1787828369.079807:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828369.079814:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828369.079827:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828369.079831:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972950272:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828369.079847:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972950528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828369.079867:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828369.079874:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828369.079878:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828369.079889:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4819 00000100:00080000:3.0:1787828369.079897:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828369.079906:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828369.079916:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828369.079943:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972950656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828369.079967:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828369.079974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828369.079978:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828369.079988:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4819 00000100:00080000:3.0:1787828369.079999:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828369.080011:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828369.080028:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828369.080045:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972950400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828369.080067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972950784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828369.080095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828369.080101:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828369.080106:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828369.080113:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4819 00000100:00080000:3.0:1787828369.080124:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828369.080131:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972950528:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828369.080134:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828369.080159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828369.080185:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972950656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828369.080231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972950912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828369.080253:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828369.080259:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828369.080263:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828369.080275:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972950784:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828369.080283:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4819 00000100:00080000:3.0:1787828369.080308:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828369.080321:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828369.080333:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828369.080434:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972950912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828369.080441:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972951040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828369.080489:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972951040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828369.591460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828369.591460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828369.847665:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828370.615512:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828370.615530:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828370.871544:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828371.639494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828371.639501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828371.895586:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828372.153417:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787828372.153483:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828372.153501:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828372.153538:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828372.153546:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828372.153555:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828372.153564:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828372.153574:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828372.153580:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787828372.153611:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828372] req@ffff9c1a7895fc00 x1874668972951040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828372.153674:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972951040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828372.153688:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828372.153692:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828372.153696:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972951040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828372.153702:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828372] req@ffff9c1a7895e940 x1874668972950912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828372.153710:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972950912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828372.153716:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828372.153719:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828372.153721:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972950912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828372.153725:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828372] req@ffff9c1a7895c3c0 x1874668972950528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828372.153733:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972950528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828372.153737:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828372.153739:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828372.153741:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972950528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828372.153745:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828372] req@ffff9c1a7895cf00 x1874668972950656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828372.153752:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972950656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828372.153758:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828372.153760:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828372.153763:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972950656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828372.153767:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828372] req@ffff9c1a7895f480 x1874668972950784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828372.153773:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972950784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828372.153778:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828372.153780:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828372.153782:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972950784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828372.153786:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828372] req@ffff9c1a7895d2c0 x1874668972950272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828372.153793:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972950272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828372.153797:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828372.153800:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828372.153802:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972950272:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828372.153805:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828369/real 1787828372] req@ffff9c1a7895c000 x1874668972950400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828372.153812:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972950400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828372.153818:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828372.153820:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828372.153822:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972950400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828372.663464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828372.663477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828372.919558:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828373.687556:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828373.687556:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828373.943690:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828374.199475:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828374.199486:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828374.199490:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828374.199501:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4824 00000100:00080000:3.0:1787828374.199511:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828374.199519:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828374.199559:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972951168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828374.199589:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828374.199593:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828374.199594:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828374.199598:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4824 00000100:00080000:3.0:1787828374.199601:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828374.199604:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828374.199613:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828374.199625:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972951296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828374.199634:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828374.199636:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828374.199638:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828374.199642:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4824 00000100:00080000:3.0:1787828374.199645:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828374.199648:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828374.199651:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828374.199659:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972951424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828374.199668:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828374.199671:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828374.199672:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828374.199687:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4824 00000100:00080000:3.0:1787828374.199690:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828374.199697:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828374.199703:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828374.199705:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972951168:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828374.199716:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972951552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828374.199726:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828374.199728:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828374.199730:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828374.199733:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4824 00000100:00080000:3.0:1787828374.199737:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828374.199739:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828374.199743:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828374.199755:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972951680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828374.199764:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828374.199767:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828374.199770:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828374.199776:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4824 00000100:00080000:3.0:1787828374.199781:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828374.199787:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787828374.199792:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828374.199797:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828374.199819:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972951808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828374.199833:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828374.199838:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828374.199841:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828374.199847:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4824 00000100:00100000:1.0:1787828374.199850:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828374] req@ffff9c1a7895e580 x1874668972951168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828374.199853:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828374.199863:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828374.199887:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972951168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828374.199887:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828374.199907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828374.199913:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972951936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828374.199916:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828374.199927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972951168:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828374.199951:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972951296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828374.199987:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828374.200004:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828374] req@ffff9c1a7895c000 x1874668972951296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828374.200028:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972951296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828374.200041:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828374.200046:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828374.200052:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972951296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828374.200061:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972951424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828374.200096:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828374.200118:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828374] req@ffff9c1a7895d2c0 x1874668972951424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828374.200129:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972951424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828374.200137:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828374.200140:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828374.200142:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972951424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828374.200194:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972951552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828374.200221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828374.200237:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828374] req@ffff9c1a7895f480 x1874668972951552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828374.200253:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972951552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828374.200264:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828374.200268:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828374.200272:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972951552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828374.200279:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972951680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828374.200302:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828374.200317:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828374] req@ffff9c1a7895cf00 x1874668972951680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828374.200331:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972951680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828374.200341:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828374.200390:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828374.200396:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972951680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828374.200403:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972951808:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828374.200431:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828374.200447:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828374] req@ffff9c1a7895c3c0 x1874668972951808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828374.200463:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972951808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828374.200474:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828374.200478:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828374.200483:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972951808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828374.200491:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972951936:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828374.200518:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828374.200610:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828374/real 1787828374] req@ffff9c1a7895e940 x1874668972951936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828374.200648:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972951936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828374.200662:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828374.200667:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828374.200672:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972951936:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828374.711491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828374.711571:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828374.967696:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828375.735451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828375.735502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828375.991670:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828376.759527:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828376.759567:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828377.015766:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828377.783432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828377.783432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828378.039526:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828378.807442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828378.807442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828379.063511:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828379.319438:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828379.319460:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828379.319464:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828379.319476:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4829 00000100:00080000:3.0:1787828379.319488:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828379.319495:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828379.319554:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972952064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828379.319583:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828379.319586:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828379.319589:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828379.319593:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4829 00000100:00080000:3.0:1787828379.319596:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828379.319617:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828379.319625:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828379.319636:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972952192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828379.319645:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828379.319647:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828379.319649:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828379.319651:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4829 00000100:00080000:3.0:1787828379.319654:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828379.319656:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828379.319660:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828379.319663:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972952064:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828379.319672:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972952320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828379.319682:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828379.319685:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828379.319686:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828379.319690:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4829 00000100:00080000:3.0:1787828379.319692:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828379.319694:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828379.319697:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828379.319716:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972952448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828379.319723:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828379.319725:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828379.319727:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828379.319730:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4829 00000100:00080000:3.0:1787828379.319733:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828379.319735:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828379.319738:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828379.319747:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972952576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828379.319753:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828379.319755:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828379.319756:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828379.319762:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828379.319763:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4829 00000100:00080000:3.0:1787828379.319768:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828379.319783:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828379.319787:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828379.319797:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972952704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828379.319803:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828379.319806:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828379.319807:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828379.319811:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4829 00000100:00080000:3.0:1787828379.319814:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828379.319817:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828379.319821:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828379.319825:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff9c1a7895fc00 x1874668972952064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828379.319832:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972952832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828379.319871:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972952064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828379.319886:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828379.319892:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828379.319897:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972952064:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828379.319906:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972952192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828379.319934:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828379.319949:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff9c1a7895de00 x1874668972952192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828379.319973:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972952192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828379.319980:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828379.319982:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828379.319986:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972952192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828379.320006:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972952320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828379.320034:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828379.320042:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff9c1a7895c780 x1874668972952320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828379.320051:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972952320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828379.320057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828379.320059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828379.320061:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972952320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828379.320065:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972952448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828379.320076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828379.320083:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff9c1a7895f840 x1874668972952448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828379.320100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972952448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828379.320106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828379.320108:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828379.320110:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972952448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828379.320114:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972952576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828379.320146:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828379.320164:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff9c1a7895da40 x1874668972952576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828379.320173:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972952576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828379.320179:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828379.320182:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828379.320184:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972952576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828379.320188:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972952704:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828379.320203:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828379.320210:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff9c1a7895cb40 x1874668972952704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828379.320229:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972952704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828379.320235:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828379.320237:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828379.320239:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972952704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828379.320243:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972952832:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828379.320254:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828379.320260:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828379/real 1787828379] req@ffff9c1a7895f0c0 x1874668972952832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828379.320268:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972952832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828379.320275:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828379.320277:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828379.320280:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972952832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828379.831417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828379.831417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828380.087531:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828380.855485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828380.855485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828381.111608:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828381.879499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828381.879505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828382.135462:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828382.903453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828382.903453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828383.159455:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828383.927490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828383.927557:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828384.183602:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828384.439440:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828384.439457:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828384.439465:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.439482:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4834 00000100:00080000:3.0:1787828384.439517:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.439532:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828384.439599:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972952960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828384.439643:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828384.439650:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828384.439654:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.439660:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4834 00000100:00080000:3.0:1787828384.439667:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.439673:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828384.439686:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828384.439708:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972953088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828384.439730:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828384.439740:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828384.439742:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972952960:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828384.439746:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.439767:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4834 00000100:00080000:3.0:1787828384.439772:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.439777:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828384.439784:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828384.439806:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972953216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828384.439825:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828384.439832:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828384.439836:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.439844:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4834 00000100:00080000:3.0:1787828384.439866:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.439876:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828384.439887:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828384.439917:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972953344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828384.439929:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828384.440249:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828384.440253:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828384.440254:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.440259:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4834 00000100:00080000:3.0:1787828384.440263:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.440267:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828384.440273:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828384.440327:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972953472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828384.440335:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828384.440337:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828384.440338:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.440341:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4834 00000100:00080000:3.0:1787828384.440364:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.440368:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828384.440372:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828384.440381:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972953600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828384.440389:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828384.440391:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828384.440393:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828384.440397:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4834 00000100:00080000:3.0:1787828384.440399:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828384.440402:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828384.440405:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828384.440411:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972953728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828384.441162:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff9c1a7895e1c0 x1874668972952960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.441179:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972952960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.441189:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.441193:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828384.441196:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972952960:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828384.441221:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972953088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828384.441242:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828384.441252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff9c1a7895f0c0 x1874668972953088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.441260:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972953088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.441266:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.441269:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828384.441285:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972953088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828384.441290:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972953216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828384.441304:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828384.441312:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff9c1a7895cb40 x1874668972953216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.441320:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972953216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.441326:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.441328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828384.441330:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972953216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828384.441334:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972953344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828384.441362:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828384.441372:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff9c1a7895da40 x1874668972953344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.441380:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972953344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.441387:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.441390:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828384.441392:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972953344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828384.441396:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972953472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828384.441409:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828384.441416:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff9c1a7895f840 x1874668972953472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.441423:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972953472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.441429:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.441430:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828384.441432:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972953472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828384.441436:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972953600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828384.441448:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828384.441454:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff9c1a7895c780 x1874668972953600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.441461:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972953600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.441466:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.441468:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828384.441470:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972953600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828384.441474:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972953728:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828384.441484:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828384.441489:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828384/real 1787828384] req@ffff9c1a7895de00 x1874668972953728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828384.441497:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972953728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828384.441503:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828384.441505:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828384.441507:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972953728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828384.951497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828384.951565:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828385.207611:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828385.975401:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828385.975460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828386.231501:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828386.999420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828386.999507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828387.255589:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828388.023447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828388.023503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828388.279637:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828389.047588:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828389.047591:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828389.303604:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828389.559451:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828389.559463:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828389.559469:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828389.559484:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4839 00000100:00080000:3.0:1787828389.559514:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828389.559534:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828389.559580:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972953856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828389.559607:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828389.559610:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828389.559612:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828389.559616:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4839 00000100:00080000:3.0:1787828389.559618:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828389.559621:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828389.559633:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828389.559643:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972953984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828389.559673:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828389.559680:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828389.559682:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972953856:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828389.559684:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828389.559698:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4839 00000100:00080000:3.0:1787828389.559703:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828389.559707:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828389.559713:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828389.559728:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972954112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828389.559740:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828389.559745:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828389.559748:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828389.559751:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4839 00000100:00080000:3.0:1787828389.559756:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828389.559760:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828389.559764:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828389.559777:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972954240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828389.559787:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828389.559792:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828389.559811:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828389.559815:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4839 00000100:00080000:3.0:1787828389.559820:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828389.559828:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828389.559829:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828389.559838:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828389.559855:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972954368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828389.559890:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828389.559894:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828389.559897:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828389.559904:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4839 00000100:00080000:3.0:1787828389.559910:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828389.559914:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828389/real 1787828389] req@ffff9c1a7895fc00 x1874668972953856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828389.559919:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828389.559933:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828389.559956:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972954496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828389.559970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972953856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828389.559972:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828389.559983:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828389.559987:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828389.560001:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4839 00000100:00080000:3.0:1787828389.560009:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828389.560013:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828389.560014:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828389.560027:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828389.560028:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828389.560038:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972953856:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828389.560046:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972954624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828389.560110:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972953984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828389.560148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828389.560164:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828389/real 1787828389] req@ffff9c1a7895e940 x1874668972953984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828389.560198:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972953984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828389.560208:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828389.560212:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828389.560231:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972953984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828389.560237:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972954112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828389.560252:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828389.560260:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828389/real 1787828389] req@ffff9c1a7895c3c0 x1874668972954112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828389.560269:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972954112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828389.560275:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828389.560277:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828389.560279:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972954112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828389.560283:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972954240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828389.560319:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828389.560336:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828389/real 1787828389] req@ffff9c1a7895cf00 x1874668972954240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828389.560357:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972954240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828389.560422:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828389.560427:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828389.560431:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972954240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828389.560439:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972954368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828389.560466:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828389.560480:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828389/real 1787828389] req@ffff9c1a7895f480 x1874668972954368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828389.560509:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972954368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828389.560516:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828389.560518:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828389.560520:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972954368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828389.560524:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972954496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828389.560544:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828389.560551:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828389/real 1787828389] req@ffff9c1a7895d2c0 x1874668972954496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828389.560560:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972954496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828389.560565:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828389.560568:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828389.560571:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972954496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828389.560575:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972954624:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828389.560586:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828389.560592:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828389/real 1787828389] req@ffff9c1a7895c000 x1874668972954624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828389.560600:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972954624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828389.560606:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828389.560608:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828389.560610:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972954624:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828390.071492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828390.071493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828390.327569:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828391.095483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828391.095485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828391.351608:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828392.119523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828392.119541:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828392.375574:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828393.143515:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828393.143529:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828393.399581:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828394.167477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828394.167507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828394.423640:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828394.679613:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828394.679626:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828394.679632:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828394.679645:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4845 00000100:00080000:0.0:1787828394.679659:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828394.679669:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828394.679718:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972954752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828394.679760:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828394.679765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828394.679768:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828394.679773:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4845 00000100:00080000:0.0:1787828394.679778:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828394.679784:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828394.679797:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828394.679819:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972954880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828394.679843:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972954752:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828394.679844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828394.679875:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828394.679881:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828394.679892:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4845 00000100:00080000:0.0:1787828394.679903:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828394.679914:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828394.679926:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828394.679957:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972955008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828394.679980:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828394.679985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828394.679988:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828394.680000:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4845 00000800:00000100:2.0:1787828394.680002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828394.680014:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828394.680049:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828394.680059:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828394.680078:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972955136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828394.680093:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828394.680098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828394.680102:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828394.680109:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828394/real 1787828394] req@ffff9c1a78860000 x1874668972954752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828394.680114:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4845 00000100:00080000:0.0:1787828394.680129:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828394.680149:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828394.680157:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828394.680165:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972954752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828394.680178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972955264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828394.680204:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828394.680208:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828394.680216:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828394.680219:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828394.680227:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828394.680230:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972954752:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828394.680241:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4845 00000100:00080000:0.0:1787828394.680254:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828394.680263:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828394.680273:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828394.680289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972955392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828394.680295:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972954880:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828394.680328:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828394.680334:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828394.680336:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828394.680409:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4845 00000100:00080000:0.0:1787828394.680414:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828394.680425:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828394.680427:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787828394.680451:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828394.680491:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828394/real 1787828394] req@ffff9c1a78863c00 x1874668972954880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828394.680492:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972955520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828394.680620:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972954880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828394.680635:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828394.680642:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828394.680648:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972954880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828394.680658:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972955008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828394.680684:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828394.680696:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828394/real 1787828394] req@ffff9c1a788630c0 x1874668972955008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828394.680709:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972955008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828394.680718:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828394.680721:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828394.680724:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972955008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828394.680731:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972955136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828394.680750:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828394.680759:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828394/real 1787828394] req@ffff9c1a78861680 x1874668972955136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828394.680772:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972955136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828394.680781:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828394.680784:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828394.680787:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972955136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828394.680794:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972955264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828394.680811:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828394.680820:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828394/real 1787828394] req@ffff9c1a78862580 x1874668972955264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828394.680832:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972955264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828394.680841:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828394.680844:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828394.680847:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972955264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828394.680872:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972955392:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828394.680891:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828394.680901:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828394/real 1787828394] req@ffff9c1a78861e00 x1874668972955392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828394.680921:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972955392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828394.680931:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828394.680934:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828394.680937:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972955392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828394.680943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972955520:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828394.680959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828394.680968:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828394/real 1787828394] req@ffff9c1a78860780 x1874668972955520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828394.680980:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972955520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828394.680990:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828394.680993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828394.680996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972955520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828395.191448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828395.191454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828395.447615:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828396.215462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828396.215462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828396.471661:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828397.239476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828397.239500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828397.495675:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828398.263494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828398.263518:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828398.519580:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828399.287451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828399.287497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828399.543535:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828399.799476:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828399.799491:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828399.799498:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828399.799510:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4850 00000100:00080000:3.0:1787828399.799522:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828399.799529:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828399.799572:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972955648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828399.799601:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828399.799620:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828399.799621:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828399.799624:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4850 00000100:00080000:3.0:1787828399.799627:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828399.799629:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828399.799637:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828399.799683:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972955776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828399.799693:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972955648:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828399.799702:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828399.799707:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828399.799709:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828399.799716:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4850 00000100:00080000:3.0:1787828399.799733:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828399.799738:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828399.799744:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828399.799755:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972955904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828399.799764:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828399.799768:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1787828399.799770:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828399.799771:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828399.799778:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4850 00000100:00080000:3.0:1787828399.799782:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828399.799785:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828399.799790:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828399.799811:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972956032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828399.799834:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828399.799836:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828399.799838:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828399.799842:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828399/real 1787828399] req@ffff9c1a7895e580 x1874668972955648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828399.799844:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4850 00000100:00080000:3.0:1787828399.799849:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828399.799855:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828399.799862:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828399.799879:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972955648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828399.799901:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972956160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828399.799934:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828399.799950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828399.799953:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828399.799961:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828399.799962:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972955648:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828399.799965:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828399.800017:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4850 00000100:00080000:3.0:1787828399.800021:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828399.800031:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828399.800035:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828399.800044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972956288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828399.800049:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972955776:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828399.800057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828399.800060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828399.800062:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828399.800065:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4850 00000100:00080000:3.0:1787828399.800075:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828399.800076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828399.800080:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828399.800083:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828399.800096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972956416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828399.800101:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828399/real 1787828399] req@ffff9c1a7895c000 x1874668972955776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828399.800115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972955776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828399.800123:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828399.800126:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828399.800128:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972955776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828399.800133:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972955904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828399.800146:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828399.800152:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828399/real 1787828399] req@ffff9c1a7895d2c0 x1874668972955904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828399.800160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972955904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828399.800170:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828399.800172:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828399.800175:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972955904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828399.800178:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972956032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828399.800188:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828399.800194:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828399/real 1787828399] req@ffff9c1a7895f480 x1874668972956032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828399.800200:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972956032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828399.800206:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828399.800208:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828399.800210:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972956032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828399.800213:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972956160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828399.800223:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828399.800228:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828399/real 1787828399] req@ffff9c1a7895cf00 x1874668972956160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828399.800236:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972956160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828399.800241:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828399.800242:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828399.800245:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972956160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828399.800258:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972956288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828399.800279:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828399.800285:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828399/real 1787828399] req@ffff9c1a7895c3c0 x1874668972956288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828399.800293:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972956288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828399.800299:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828399.800300:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828399.800302:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972956288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828399.800305:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972956416:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828399.800315:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828399.800322:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828399/real 1787828399] req@ffff9c1a7895e940 x1874668972956416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828399.800335:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972956416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828399.800370:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828399.800375:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828399.800379:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972956416:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828400.311495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828400.311511:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828400.567537:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828401.335431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828401.335446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828401.591589:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828402.359492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828402.359492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828402.615585:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828403.383459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828403.383497:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828403.639515:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828404.407457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828404.407462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828404.663626:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828404.919459:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828404.919471:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828404.919477:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828404.919488:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4855 00000100:00080000:3.0:1787828404.919501:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828404.919508:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828404.919561:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972956544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828404.919590:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828404.919593:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828404.919595:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828404.919598:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4855 00000100:00080000:3.0:1787828404.919600:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828404.919604:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828404.919614:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828404.919625:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972956672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828404.919633:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828404.919636:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828404.919637:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828404.919640:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4855 00000100:00080000:3.0:1787828404.919642:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828404.919647:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972956544:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828404.919647:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828404.919654:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828404.919671:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972956800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828404.919679:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828404.919682:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828404.919683:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828404.919686:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4855 00000100:00080000:3.0:1787828404.919689:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828404.919692:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828404.919696:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828404.919704:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972956928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828404.919720:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828404.919723:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828404.919724:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828404.919727:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4855 00000100:00080000:3.0:1787828404.919732:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828404.919734:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828404.919737:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828404.919746:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972957056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828404.919749:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828404.919758:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828404.919761:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828404.919762:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828404.919767:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4855 00000100:00080000:3.0:1787828404.919769:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828404.919772:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828404.919775:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828404.919782:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972957184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828404.919788:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828404.919790:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828404.919791:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828404.919794:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4855 00000100:00080000:3.0:1787828404.919797:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828404.919801:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828404.919804:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828404.919806:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828404/real 1787828404] req@ffff9c1a7895fc00 x1874668972956544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828404.919815:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972957312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828404.919830:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972956544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828404.919840:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828404.919844:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828404.919848:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972956544:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828404.919864:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972956672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828404.919883:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828404.919893:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828404/real 1787828404] req@ffff9c1a7895de00 x1874668972956672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828404.919901:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972956672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828404.919907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828404.919909:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828404.919911:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972956672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828404.919935:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972956800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828404.919963:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828404.919972:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828404/real 1787828404] req@ffff9c1a7895c780 x1874668972956800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828404.919981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972956800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828404.919987:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828404.919989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828404.919991:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972956800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828404.920004:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972956928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828404.920016:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828404.920022:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828404/real 1787828404] req@ffff9c1a7895f840 x1874668972956928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828404.920029:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972956928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828404.920035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828404.920036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828404.920038:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972956928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828404.920042:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972957056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828404.920053:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828404.920058:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828404/real 1787828404] req@ffff9c1a7895da40 x1874668972957056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828404.920088:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972957056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828404.920094:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828404.920096:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828404.920098:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972957056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828404.920102:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972957184:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828404.920115:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828404.920121:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828404/real 1787828404] req@ffff9c1a7895cb40 x1874668972957184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828404.920132:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972957184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828404.920140:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828404.920144:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828404.920148:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972957184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828404.920154:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972957312:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828404.920174:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828404.920186:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828404/real 1787828404] req@ffff9c1a7895f0c0 x1874668972957312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828404.920205:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972957312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828404.920215:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828404.920217:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828404.920220:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972957312:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828405.431476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828405.431476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828405.687599:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828406.455538:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828406.455598:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828406.711512:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828407.479441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828407.479505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828407.735584:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828408.503511:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828408.503583:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828408.759676:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828409.527578:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828409.527604:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828409.783642:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828410.039483:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828410.039496:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828410.039502:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828410.039515:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4860 00000100:00080000:3.0:1787828410.039530:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828410.039538:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828410.039587:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972957440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828410.039630:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828410.039635:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828410.039638:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828410.039643:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4860 00000100:00080000:3.0:1787828410.039647:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828410.039655:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828410.039671:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828410.039711:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972957568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828410.039734:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828410.039741:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828410.039746:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828410.039783:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4860 00000100:00080000:3.0:1787828410.039794:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828410.039803:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828410.039820:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972957440:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787828410.039822:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828410.039863:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972957696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828410.039888:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828410.039893:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828410.039896:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828410.039901:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4860 00000100:00080000:3.0:1787828410.039906:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828410.039942:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828410.039949:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828410.039969:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972957824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828410.039981:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828410.039985:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828410.039997:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828410.040002:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4860 00000100:00080000:3.0:1787828410.040010:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828410.040015:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828410.040020:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828410.040033:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972957952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828410.040064:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828410.040068:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828410.040071:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828410.040079:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828410.040086:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4860 00000100:00080000:3.0:1787828410.040090:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828410.040096:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828410.040108:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828410.040131:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972958080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828410.040173:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828410.040181:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828410.040186:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828410.040190:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff9c1a7895e1c0 x1874668972957440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828410.040221:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4860 00000100:00080000:3.0:1787828410.040240:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828410.040246:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972957440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828410.040256:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828410.040285:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828410.040292:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828410.040302:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828410.040310:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972958208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828410.040322:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972957440:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828410.040461:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972957568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828410.040539:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828410.040563:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff9c1a7895f0c0 x1874668972957568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828410.040590:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972957568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828410.040601:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828410.040605:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828410.040609:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972957568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828410.040650:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972957696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828410.040694:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828410.040706:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff9c1a7895cb40 x1874668972957696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828410.040719:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972957696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828410.040727:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828410.040730:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828410.040733:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972957696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828410.040741:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972957824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828410.040777:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828410.040790:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff9c1a7895da40 x1874668972957824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828410.040805:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972957824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828410.040825:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828410.040829:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828410.040833:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972957824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828410.040840:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972957952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828410.040861:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828410.040872:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff9c1a7895f840 x1874668972957952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828410.040886:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972957952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828410.040896:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828410.040900:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828410.040904:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972957952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828410.040955:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972958080:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828410.040990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828410.041029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff9c1a7895c780 x1874668972958080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828410.041055:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972958080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828410.041087:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828410.041092:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828410.041097:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972958080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828410.041104:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972958208:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828410.041127:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828410.041139:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828410/real 1787828410] req@ffff9c1a7895de00 x1874668972958208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828410.041151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972958208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828410.041159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828410.041162:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828410.041165:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972958208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828410.551463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828410.551508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828410.807724:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828411.575522:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828411.575523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828411.831712:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828412.599457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828412.599507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828412.855736:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828413.623525:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828413.623525:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828413.879690:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828414.647473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828414.647475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828414.903580:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828415.159517:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828415.159526:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828415.159531:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828415.159542:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4865 00000100:00080000:0.0:1787828415.159553:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828415.159561:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828415.159610:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972958336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828415.159640:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828415.159644:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828415.159646:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828415.159650:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4865 00000100:00080000:0.0:1787828415.159653:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828415.159657:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828415.159666:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828415.159677:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972958464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828415.159687:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828415.159691:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828415.159693:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828415.159697:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4865 00000100:00080000:0.0:1787828415.159699:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828415.159702:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828415.159707:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828415.159715:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972958592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828415.159725:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828415.159728:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828415.159730:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828415.159734:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4865 00000100:00080000:0.0:1787828415.159737:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828415.159740:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828415.159744:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828415.159752:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972958720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828415.159762:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828415.159765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828415.159767:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828415.159773:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4865 00000100:00080000:0.0:1787828415.159783:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828415.159785:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972958336:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828415.159791:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828415.159805:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828415.159819:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972958848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828415.159830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828415.159833:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828415.159835:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828415.159840:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4865 00000100:00080000:0.0:1787828415.159844:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828415.159850:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828415.159856:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828415.159869:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972958976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828415.159897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828415.159900:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828415.159902:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828415.159907:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4865 00000100:00080000:0.0:1787828415.159911:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828415.159913:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828415.159930:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828415.159945:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972959104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828415.159945:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828415.160012:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff9c1a788621c0 x1874668972958336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828415.160035:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972958336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828415.160048:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828415.160054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828415.160059:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972958336:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828415.160069:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972958464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828415.160107:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828415.160120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff9c1a78862940 x1874668972958464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828415.160133:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972958464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828415.160141:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828415.160145:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828415.160148:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972958464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828415.160155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972958592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828415.160172:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828415.160181:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff9c1a78861a40 x1874668972958592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828415.160193:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972958592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828415.160201:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828415.160203:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828415.160206:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972958592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828415.160241:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972958720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828415.160259:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828415.160268:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff9c1a78860f00 x1874668972958720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828415.160292:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972958720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828415.160301:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828415.160305:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828415.160308:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972958720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828415.160314:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972958848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828415.160337:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828415.160400:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff9c1a78862d00 x1874668972958848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828415.160419:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972958848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828415.160432:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828415.160438:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828415.160443:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972958848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828415.160454:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972958976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828415.160483:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828415.160496:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff9c1a78860b40 x1874668972958976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828415.160523:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972958976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828415.160536:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828415.160541:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828415.160547:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972958976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828415.160557:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972959104:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828415.160587:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828415.160600:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828415/real 1787828415] req@ffff9c1a78863840 x1874668972959104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828415.160612:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972959104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828415.160624:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828415.160629:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828415.160635:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972959104:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828415.671469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828415.671485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828415.927609:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828416.695518:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828416.695574:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828416.951565:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828417.719502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828417.719522:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828417.975758:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828418.743458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828418.743458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828418.999662:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828419.767478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828419.767548:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828420.023683:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828420.279588:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828420.279604:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828420.279612:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828420.279637:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4870 00000100:00080000:3.0:1787828420.279653:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828420.279663:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828420.279761:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972959232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828420.279800:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828420.279805:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828420.279807:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828420.279815:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4870 00000100:00080000:3.0:1787828420.279822:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828420.279829:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828420.279865:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828420.279867:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972959232:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828420.279917:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972959360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828420.279932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828420.279936:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828420.279939:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828420.279944:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4870 00000100:00080000:3.0:1787828420.279950:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828420.279955:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828420.279961:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828420.279976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972959488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828420.279988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828420.279995:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787828420.279997:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828420.279999:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828420.280007:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4870 00000100:00080000:3.0:1787828420.280013:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828420.280018:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828420.280030:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828420.280064:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff9c1a7895fc00 x1874668972959232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828420.280088:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972959232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.280118:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.280123:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828420.280128:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972959232:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828420.280180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972959616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828420.280185:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972959360:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828420.280214:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828420.280221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828420.280225:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828420.280230:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828420.280238:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4870 00000100:00080000:3.0:1787828420.280254:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828420.280265:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff9c1a7895e940 x1874668972959360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828420.280267:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828420.280300:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828420.280308:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972959360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.280320:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.280325:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828420.280330:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972959360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828420.280337:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972959488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828420.280357:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828420.280367:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff9c1a7895c3c0 x1874668972959488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828420.280378:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972959488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.280387:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.280436:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828420.280440:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972959488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828420.280461:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972959616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828420.280484:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828420.280498:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff9c1a7895cf00 x1874668972959616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828420.280512:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972959616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.280540:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.280546:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828420.280549:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972959616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828420.280618:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972959744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828420.280678:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972959744:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828420.280726:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828420.280732:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828420.280736:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828420.280738:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828420.280749:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4870 00000100:00080000:3.0:1787828420.280775:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828420.280782:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828420.280790:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff9c1a7895f480 x1874668972959744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828420.280792:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828420.280815:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972959744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828420.280860:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972959872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828420.280870:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.280886:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828420.280889:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828420.280911:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972959744:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828420.280913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828420.280927:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828420.280937:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4870 00000100:00080000:3.0:1787828420.280944:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828420.280950:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828420.280955:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972959872:192.168.203.148@tcp:8:kworker.0 00000008:00100000:3.0:1787828420.280960:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828420.280981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972960000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828420.281001:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828420.281027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff9c1a7895d2c0 x1874668972959872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828420.281051:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972959872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.281064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.281068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828420.281072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972959872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828420.281104:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972960000:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828420.281127:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828420.281138:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828420/real 1787828420] req@ffff9c1a7895c000 x1874668972960000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828420.281149:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972960000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828420.281176:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828420.281180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828420.281183:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972960000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828420.791462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828420.791472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828421.047570:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828421.815457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828421.815457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828422.071560:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828422.839468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828422.839495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828423.095528:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828423.863458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828423.863548:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828424.119579:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828424.887420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828424.887441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828425.143553:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828425.399502:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828425.399517:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828425.399534:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828425.399549:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4875 00000100:00080000:3.0:1787828425.399565:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828425.399577:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828425.399632:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972960128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828425.399691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828425.399698:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828425.399701:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828425.399708:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4875 00000100:00080000:3.0:1787828425.399714:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828425.399723:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828425.399738:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828425.399760:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972960128:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828425.399770:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972960256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828425.399853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828425.399860:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828425.399863:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828425.399869:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4875 00000100:00080000:3.0:1787828425.399877:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828425.399882:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828425.399890:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828425.399906:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828425.399913:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972960384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828425.399946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828425.399953:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828425.399957:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828425.399965:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4875 00000100:00080000:3.0:1787828425.399973:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828425.399983:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828425.399993:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828425.399995:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff9c1a7895e580 x1874668972960128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828425.400027:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972960512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828425.400030:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972960128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828425.400096:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828425.400100:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828425.400110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828425.400112:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828425.400119:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828425.400123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972960128:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828425.400137:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4875 00000100:00080000:3.0:1787828425.400148:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828425.400157:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828425.400163:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972960256:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787828425.400173:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828425.400203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972960640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828425.400209:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828425.400230:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828425.400237:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff9c1a7895c000 x1874668972960256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828425.400238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828425.400245:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828425.400259:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4875 00000100:00080000:3.0:1787828425.400272:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828425.400282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972960256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828425.400283:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828425.400296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828425.400312:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828425.400320:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828425.400323:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972960768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828425.400330:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972960256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828425.400353:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972960384:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828425.400357:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828425.400366:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828425.400369:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828425.400409:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4875 00000100:00080000:3.0:1787828425.400415:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828425.400420:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828425.400426:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828425.400442:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972960896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828425.400458:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828425.400500:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff9c1a7895d2c0 x1874668972960384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828425.400526:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972960384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828425.400541:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828425.400548:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828425.400553:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972960384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828425.400566:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972960512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828425.400605:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828425.400626:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff9c1a7895f480 x1874668972960512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828425.400651:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972960512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828425.400675:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828425.400681:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828425.400686:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972960512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828425.400718:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972960640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828425.400753:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828425.400769:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff9c1a7895cf00 x1874668972960640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828425.400783:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972960640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828425.400793:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828425.400797:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828425.400800:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972960640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828425.400840:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972960768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828425.400863:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828425.400875:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff9c1a7895c3c0 x1874668972960768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828425.400890:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972960768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828425.400919:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828425.400922:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828425.400926:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972960768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828425.400933:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972960896:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828425.400954:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828425.400966:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828425/real 1787828425] req@ffff9c1a7895e940 x1874668972960896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828425.400980:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972960896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828425.400990:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828425.400993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828425.400997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972960896:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828425.911420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828425.911498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828426.167764:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828426.935560:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828426.935579:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828427.191615:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828427.959463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828427.959466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828428.215552:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828428.983552:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828428.983553:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828429.239547:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828430.007451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828430.007452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828430.263591:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828430.519501:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828430.519518:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828430.519526:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828430.519543:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4880 00000100:00080000:3.0:1787828430.519560:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828430.519572:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828430.519621:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972961024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828430.519667:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828430.519674:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828430.519677:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828430.519686:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4880 00000100:00080000:3.0:1787828430.519695:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828430.519703:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828430.519719:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828430.519725:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972961024:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828430.519748:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972961152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828430.519803:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828430.519811:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828430.519816:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828430.519842:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4880 00000100:00080000:3.0:1787828430.519852:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828430.519860:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787828430.519870:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828430.519879:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828430.519913:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972961280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828430.519932:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828430/real 1787828430] req@ffff9c1a7895fc00 x1874668972961024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828430.519943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828430.519951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828430.519959:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828430.519975:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4880 00000100:00080000:3.0:1787828430.519983:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828430.519985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972961024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828430.520001:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787828430.520031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828430.520034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828430.520041:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828430.520050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972961024:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828430.520065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972961408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828430.520087:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972961152:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828430.520091:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828430.520102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828430.520106:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828430.520117:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4880 00000100:00080000:3.0:1787828430.520127:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828430.520132:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828430.520141:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828430.520153:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828430.520156:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828430/real 1787828430] req@ffff9c1a7895de00 x1874668972961152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828430.520184:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972961536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828430.520202:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972961152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828430.520220:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828430.520224:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828430.520236:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828430.520237:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828430.520245:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972961152:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828430.520248:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828430.520260:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972961280:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828430.520266:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4880 00000100:00080000:3.0:1787828430.520279:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828430.520289:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787828430.520298:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828430.520307:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828430.520317:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828430/real 1787828430] req@ffff9c1a7895c780 x1874668972961280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828430.520339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972961664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828430.520340:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972961280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828430.520363:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828430.520370:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828430.520374:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828430.520396:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828430.520405:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828430.520412:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972961280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828430.520447:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972961408:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828430.520451:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4880 00000100:00080000:3.0:1787828430.520466:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828430.520476:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828430.520488:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828430.520503:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828430.520520:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972961792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828430.520529:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828430/real 1787828430] req@ffff9c1a7895f840 x1874668972961408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828430.520577:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972961408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828430.520591:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828430.520597:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828430.520605:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972961408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828430.520619:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972961536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828430.520659:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828430.520680:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828430/real 1787828430] req@ffff9c1a7895da40 x1874668972961536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828430.520702:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972961536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828430.520713:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828430.520717:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828430.520721:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972961536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828430.520728:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972961664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828430.520756:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828430.520795:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828430/real 1787828430] req@ffff9c1a7895cb40 x1874668972961664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828430.520830:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972961664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828430.520841:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828430.520845:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828430.520849:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972961664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828430.520870:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972961792:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828430.520898:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828430.520911:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828430/real 1787828430] req@ffff9c1a7895f0c0 x1874668972961792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828430.520926:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972961792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828430.520936:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828430.520940:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828430.520944:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972961792:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828431.031462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828431.031463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828431.287598:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828432.055456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828432.055500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828432.311567:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828433.079424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828433.079441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828433.335548:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828434.103458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828434.103497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828434.359667:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828435.127530:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828435.127554:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828435.383679:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828435.639623:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828435.639638:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828435.639644:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828435.639655:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4885 00000100:00080000:1.0:1787828435.639686:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828435.639695:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828435.639786:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668972961920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828435.639842:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828435.639846:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828435.639849:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828435.639856:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4885 00000100:00080000:1.0:1787828435.639861:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828435.639867:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828435.639879:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828435.639894:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972961920:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828435.639897:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668972962048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828435.639926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828435.639929:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828435.639933:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828435.639941:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4885 00000100:00080000:1.0:1787828435.639948:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828435.639957:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828435.639983:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828435.639998:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668972962176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828435.640014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828435.640019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828435.640022:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828435.640030:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4885 00000100:00080000:1.0:1787828435.640037:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828435.640046:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828435.640054:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828435.640060:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972962048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828435.640070:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668972962304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828435.640087:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828435.640090:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828435.640093:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828435.640097:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4885 00000100:00080000:1.0:1787828435.640104:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828435.640109:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828435.640117:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972962176:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787828435.640118:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828435.640143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668972962432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828435.640152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828435.640155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828435.640157:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828435.640161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4885 00000100:00080000:1.0:1787828435.640166:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828435.640171:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828435.640175:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972962304:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787828435.640180:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828435.640198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668972962560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828435.640209:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972962432:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828435.640241:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828435.640246:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828435.640250:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972962560:192.168.203.148@tcp:8:kworker.0 00000100:00080000:1.0:1787828435.640252:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828435.640264:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4885 00000100:00080000:1.0:1787828435.640271:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828435.640278:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828435.640288:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828435.640307:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668972962688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828435.640386:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972962688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828436.151465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828436.151501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828436.407526:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828437.175433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828437.175495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828437.431602:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828438.199447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828438.199467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828438.455633:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828438.711695:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787828438.711783:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828438.711810:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828438.711875:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828438.711895:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828438.711916:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828438.711938:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787828438.711954:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828435/real 1787828438] req@ffff9c1a78a78000 x1874668972962304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828438.711962:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828438.711990:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787828438.712010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668972962304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828438.712031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828438.712041:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828438.712049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972962304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828438.712064:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828435/real 1787828438] req@ffff9c1a78a7a580 x1874668972962432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828438.712092:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668972962432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828438.712110:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828438.712117:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828438.712125:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972962432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828438.712139:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828435/real 1787828438] req@ffff9c1a78a78f00 x1874668972962048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828438.712164:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668972962048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828438.712183:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828438.712189:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828438.712196:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972962048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828438.712210:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828435/real 1787828438] req@ffff9c1a78a7b480 x1874668972962176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828438.712233:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668972962176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828438.712249:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828438.712255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828438.712261:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972962176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828438.712274:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828435/real 1787828438] req@ffff9c1a78a79a40 x1874668972961920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828438.712299:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668972961920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828438.712315:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828438.712321:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828438.712327:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972961920:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828438.712433:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828435/real 1787828438] req@ffff9c1a78a792c0 x1874668972962688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828438.712464:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668972962688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828438.712487:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828438.712506:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828438.712515:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972962688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828438.712529:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828435/real 1787828438] req@ffff9c1a78a7b840 x1874668972962560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828438.712552:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668972962560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828438.712569:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828438.712576:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828438.712582:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972962560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828439.223510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828439.223511:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828439.479561:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828440.247442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828440.247469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828440.503611:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828440.759572:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828440.759585:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828440.759590:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828440.759603:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4891 00000100:00080000:3.0:1787828440.759615:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828440.759624:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828440.759687:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972962816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828440.759780:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828440.759787:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828440.759791:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828440.759799:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4891 00000100:00080000:3.0:1787828440.759806:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828440.759813:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828440.759828:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828440.759856:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972962944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828440.759878:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972962816:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828440.759885:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828440.759895:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828440.759898:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828440.759905:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4891 00000100:00080000:3.0:1787828440.759910:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828440.759916:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828440.759924:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828440.759944:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972963072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828440.759965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828440.759972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828440.759976:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828440.759984:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4891 00000800:00000100:0.0:1787828440.759994:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828440.759995:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828440.760009:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828440.760020:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828440.760046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972963200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828440.760066:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828440.760068:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828440/real 1787828440] req@ffff9c1a7895e1c0 x1874668972962816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828440.760079:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828440.760088:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828440.760106:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4891 00000100:00080000:3.0:1787828440.760115:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828440.760123:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828440.760130:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972962816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828440.760136:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828440.760162:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828440.760173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972963328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828440.760179:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828440.760194:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972962816:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828440.760212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828440.760220:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828440.760224:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828440.760233:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4891 00000100:00080000:3.0:1787828440.760241:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828440.760248:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972962944:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828440.760257:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828440.760272:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828440.760298:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972963456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828440.760316:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828440.760322:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828440.760326:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828440.760333:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4891 00000800:00000100:0.0:1787828440.760394:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828440.760421:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828440.760431:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828440/real 1787828440] req@ffff9c1a7895f0c0 x1874668972962944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828440.760439:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828440.760456:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828440.760475:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972962944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828440.760488:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972963584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828440.760516:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828440.760529:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828440.760552:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972962944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828440.760562:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972963072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828440.760594:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828440.760611:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828440/real 1787828440] req@ffff9c1a7895cb40 x1874668972963072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828440.760635:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972963072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828440.760648:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828440.760655:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828440.760660:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972963072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828440.760671:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972963200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828440.760719:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828440.760739:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828440/real 1787828440] req@ffff9c1a7895da40 x1874668972963200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828440.760764:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972963200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828440.760775:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828440.760779:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828440.760783:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972963200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828440.760791:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972963328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828440.760817:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828440.760830:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828440/real 1787828440] req@ffff9c1a7895f840 x1874668972963328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828440.760846:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972963328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828440.760856:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828440.760860:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828440.760864:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972963328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828440.760890:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972963456:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828440.760919:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828440.760934:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828440/real 1787828440] req@ffff9c1a7895c780 x1874668972963456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828440.760951:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972963456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828440.760971:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828440.760975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828440.760979:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972963456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828440.760988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972963584:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828440.761013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828440.761029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828440/real 1787828440] req@ffff9c1a7895de00 x1874668972963584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828440.761044:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972963584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828440.761054:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828440.761058:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828440.761062:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972963584:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828441.271485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828441.271486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828441.527620:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828442.295441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828442.295503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828442.551567:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828443.319465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828443.319465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828443.575697:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828444.343398:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828444.343430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828444.599614:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828445.367447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828445.367448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828445.623572:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828445.879516:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828445.879540:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828445.879545:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828445.879557:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4896 00000100:00080000:1.0:1787828445.879586:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828445.879592:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828445.879626:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668972963712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828445.879652:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828445.879657:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828445.879669:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828445.879674:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4896 00000100:00080000:1.0:1787828445.879678:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828445.879683:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828445.879693:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828445.879708:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668972963840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828445.879720:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828445.879723:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828445.879725:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828445.879728:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4896 00000100:00080000:1.0:1787828445.879732:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828445.879735:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828445.879739:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828445.879749:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668972963968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828445.879759:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828445.879766:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828445.879768:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828445.879770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972963712:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828445.879774:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4896 00000100:00080000:1.0:1787828445.879788:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828445.879793:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828445.879796:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828445.879812:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668972964096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828445.879823:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828445.879826:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828445.879827:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828445.879830:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4896 00000100:00080000:1.0:1787828445.879832:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828445.879834:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828445.879839:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828445.879849:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668972964224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828445.879855:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828445.879858:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828445.879860:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828445.879863:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4896 00000100:00080000:1.0:1787828445.879870:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828445.879872:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828445.879879:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828445.879900:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828445.879900:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668972964352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828445.879920:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828445.879923:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828445.879925:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828445.879930:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4896 00000100:00080000:1.0:1787828445.879932:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828445.879936:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828445.879942:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828445.879951:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668972964480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828445.879976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828445/real 1787828445] req@ffff9c1a78a7a940 x1874668972963712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828445.880001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668972963712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828445.880015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828445.880020:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828445.880027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972963712:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828445.880036:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972963840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828445.880062:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828445.880076:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828445/real 1787828445] req@ffff9c1a78a783c0 x1874668972963840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828445.880089:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668972963840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828445.880098:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828445.880102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828445.880106:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972963840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828445.880137:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972963968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828445.880156:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828445.880166:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828445/real 1787828445] req@ffff9c1a78a7a1c0 x1874668972963968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828445.880178:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668972963968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828445.880188:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828445.880191:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828445.880194:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972963968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828445.880201:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972964096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828445.880218:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828445.880228:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828445/real 1787828445] req@ffff9c1a78a78780 x1874668972964096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828445.880240:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668972964096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828445.880249:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828445.880252:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828445.880255:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972964096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828445.880261:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972964224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828445.880277:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828445.880287:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828445/real 1787828445] req@ffff9c1a78a7bc00 x1874668972964224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828445.880299:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668972964224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828445.880307:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828445.880310:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828445.880313:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972964224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828445.880319:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972964352:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828445.880336:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828445.880409:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828445/real 1787828445] req@ffff9c1a78a7b0c0 x1874668972964352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828445.880425:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668972964352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828445.880434:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828445.880438:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828445.880452:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972964352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828445.880461:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972964480:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828445.880495:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828445.880523:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828445/real 1787828445] req@ffff9c1a78a79e00 x1874668972964480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828445.880545:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668972964480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828445.880560:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828445.880565:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828445.880569:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972964480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828446.391421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828446.391427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828446.647488:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828447.415407:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828447.415437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828447.671508:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828448.439551:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828448.439557:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828448.695501:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828449.463475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828449.463482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828449.719485:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828450.487511:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828450.487591:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828450.743659:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828450.999544:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828450.999558:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828450.999565:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828450.999578:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4901 00000100:00080000:3.0:1787828450.999593:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828450.999602:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828450.999684:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972964608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828450.999725:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828450.999730:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828450.999733:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828450.999740:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4901 00000100:00080000:3.0:1787828450.999745:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828450.999750:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828450.999763:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828450.999781:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972964736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828450.999797:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828450.999806:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972964608:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828450.999807:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828450.999813:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828450.999830:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4901 00000100:00080000:3.0:1787828450.999834:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828450.999839:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828450.999845:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828450.999863:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972964864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828450.999878:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828450.999883:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828450.999886:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828450.999889:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4901 00000100:00080000:3.0:1787828450.999905:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828450.999916:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787828450.999917:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828450.999928:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828450.999948:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972964992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828450.999962:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828450.999966:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828450.999969:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828450.999976:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4901 00000100:00080000:3.0:1787828450.999982:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828450.999990:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828450/real 1787828450] req@ffff9c1a7895fc00 x1874668972964608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828450.999991:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828451.000003:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828451.000026:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972965120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828451.000037:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972964608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828451.000049:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828451.000064:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828451.000069:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828451.000082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828451.000083:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4901 00000100:00080000:3.0:1787828451.000096:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828451.000102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828451.000111:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828451.000119:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972964608:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787828451.000121:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828451.000148:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972965248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828451.000166:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828451.000170:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972964736:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828451.000172:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828451.000179:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828451.000190:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4901 00000100:00080000:3.0:1787828451.000196:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828451.000201:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828451.000210:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828451.000222:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828451.000230:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972965376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828451.000246:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828450/real 1787828451] req@ffff9c1a7895e940 x1874668972964736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828451.000308:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972964736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828451.000321:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828451.000328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828451.000333:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972964736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828451.000390:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972964864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828451.000435:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828451.000459:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828451/real 1787828451] req@ffff9c1a7895c3c0 x1874668972964864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828451.000486:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972964864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828451.000498:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828451.000502:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828451.000507:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972964864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828451.000515:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972964992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828451.000546:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828451.000560:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828451/real 1787828451] req@ffff9c1a7895cf00 x1874668972964992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828451.000575:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972964992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828451.000585:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828451.000589:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828451.000593:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972964992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828451.000599:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972965120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828451.000618:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828451.000661:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828451/real 1787828451] req@ffff9c1a7895f480 x1874668972965120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828451.000677:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972965120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828451.000688:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828451.000692:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828451.000696:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972965120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828451.000720:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972965248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828451.000747:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828451.000759:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828451/real 1787828451] req@ffff9c1a7895d2c0 x1874668972965248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828451.000783:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972965248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828451.000793:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828451.000796:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828451.000800:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972965248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828451.000806:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972965376:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828451.000825:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828451.000836:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828451/real 1787828451] req@ffff9c1a7895c000 x1874668972965376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828451.000850:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972965376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828451.000860:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828451.000863:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828451.000867:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972965376:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828451.511595:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828451.511618:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828451.767639:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828452.535553:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828452.535595:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828452.791565:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828453.559483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828453.559510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828453.815632:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828454.583547:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828454.583657:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828454.839665:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828455.607524:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828455.607578:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828455.863709:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828456.119489:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.119499:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828456.119514:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.119524:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4906 00000100:00080000:0.0:1787828456.119536:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.119544:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828456.119611:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972965504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828456.119643:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.119647:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828456.119649:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.119653:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4906 00000100:00080000:0.0:1787828456.119657:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.119662:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828456.119674:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828456.119688:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972965504:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828456.119693:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972965632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828456.119707:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.119712:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828456.119714:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.119718:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4906 00000100:00080000:0.0:1787828456.119721:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.119725:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828456.119732:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828456.119747:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972965760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828456.119758:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.119761:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1787828456.119766:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828456.119768:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.119776:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4906 00000100:00080000:0.0:1787828456.119782:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.119787:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828456.119790:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828456.119805:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972965888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828456.119813:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.119817:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828456.119818:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.119822:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4906 00000100:00080000:0.0:1787828456.119827:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828456.119828:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff9c1a788612c0 x1874668972965504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828456.119834:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828456.119847:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828456.119869:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972965504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828456.119878:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972966016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828456.119889:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.119891:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828456.119893:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.119896:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4906 00000100:00080000:0.0:1787828456.119898:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.119901:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828456.119907:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828456.119915:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972966144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828456.119925:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828456.119929:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828456.119930:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828456.119936:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4906 00000100:00080000:0.0:1787828456.119941:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828456.119944:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828456.119949:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828456.119960:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972966272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828456.119978:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.119985:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828456.119992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972965504:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828456.120017:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972965632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828456.120202:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828456.120221:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff9c1a78863840 x1874668972965632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828456.120282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972965632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828456.120298:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.120302:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828456.120307:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972965632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828456.120316:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972965760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828456.120392:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828456.120417:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff9c1a78860b40 x1874668972965760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828456.120429:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972965760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828456.120437:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.120441:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828456.120445:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972965760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828456.120453:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972965888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828456.120470:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828456.120478:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff9c1a78862d00 x1874668972965888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828456.120488:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972965888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828456.120495:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.120497:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828456.120500:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972965888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828456.120505:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972966016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828456.120518:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828456.120525:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff9c1a78860f00 x1874668972966016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828456.120534:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972966016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828456.120541:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.120543:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828456.120546:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972966016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828456.120550:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972966144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828456.120564:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828456.120571:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff9c1a78861a40 x1874668972966144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828456.120581:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972966144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828456.120606:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.120610:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828456.120613:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972966144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828456.120619:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972966272:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828456.120637:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828456.120654:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828456/real 1787828456] req@ffff9c1a78862940 x1874668972966272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828456.120664:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972966272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828456.120671:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828456.120673:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828456.120676:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972966272:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828456.631463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828456.631463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828456.887704:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828457.655542:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828457.655559:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828457.911566:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828458.679511:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828458.679511:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828458.935736:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828459.703449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828459.703449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828459.959678:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828460.727583:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828460.727626:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828460.983617:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828461.239457:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828461.239467:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828461.239471:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828461.239482:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4911 00000100:00080000:0.0:1787828461.239492:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.239498:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828461.239531:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972966400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828461.239572:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828461.239576:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828461.239578:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828461.239583:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4911 00000100:00080000:0.0:1787828461.239588:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.239593:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828461.239600:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972966400:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787828461.239602:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828461.239618:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972966528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828461.239628:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828461.239631:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828461.239633:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828461.239636:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4911 00000100:00080000:0.0:1787828461.239641:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.239645:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828461.239648:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828461.239657:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972966656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828461.239666:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828461.239670:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828461.239674:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828461.239676:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828461.239680:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4911 00000100:00080000:0.0:1787828461.239683:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.239687:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828461.239690:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828461.239700:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972966784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828461.239709:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828461.239710:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff9c1a788621c0 x1874668972966400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828461.239717:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828461.239721:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828461.239730:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4911 00000100:00080000:0.0:1787828461.239734:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.239740:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828461.239740:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972966400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828461.239744:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828461.239758:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828461.239761:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972966912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828461.239765:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828461.239773:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972966400:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828461.239781:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828461.239785:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828461.239787:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828461.239791:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4911 00000100:00080000:0.0:1787828461.239793:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.239811:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828461.239816:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828461.239818:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972966528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828461.239827:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972967040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828461.239836:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828461.239839:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828461.239840:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828461.239845:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828461.239849:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4911 00000100:00080000:0.0:1787828461.239854:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828461.239857:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828461.239878:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff9c1a78860780 x1874668972966528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828461.239879:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828461.239894:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972967168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828461.239903:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972966528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828461.239915:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828461.239918:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828461.239921:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972966528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828461.239943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972966656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828461.239972:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828461.239986:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff9c1a78861e00 x1874668972966656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828461.240000:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972966656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828461.240010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828461.240015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828461.240017:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972966656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828461.240022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972966784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828461.240039:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828461.240047:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff9c1a78862580 x1874668972966784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828461.240056:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972966784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828461.240066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828461.240068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828461.240071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972966784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828461.240074:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972966912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828461.240085:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828461.240091:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff9c1a78861680 x1874668972966912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828461.240115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972966912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828461.240121:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828461.240123:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828461.240125:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972966912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828461.240137:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972967040:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828461.240150:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828461.240156:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff9c1a788630c0 x1874668972967040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828461.240164:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972967040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828461.240169:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828461.240171:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828461.240190:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972967040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828461.240194:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972967168:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828461.240206:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828461.240212:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828461/real 1787828461] req@ffff9c1a78863c00 x1874668972967168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828461.240219:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972967168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828461.240225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828461.240227:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828461.240229:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972967168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828461.751417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828461.751425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828462.007607:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828462.775433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828462.775433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828463.031612:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828463.799415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828463.799438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828464.055633:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828464.823524:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828464.823547:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828465.079644:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828465.847486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828465.847517:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828466.103715:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828466.359472:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828466.359480:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828466.359484:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828466.359493:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4916 00000100:00080000:0.0:1787828466.359502:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828466.359509:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828466.359558:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972967296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828466.359594:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828466.359597:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828466.359598:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828466.359602:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4916 00000100:00080000:0.0:1787828466.359604:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828466.359608:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828466.359615:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828466.359627:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972967424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828466.359636:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828466.359638:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828466.359640:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828466.359646:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972967296:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828466.359647:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4916 00000100:00080000:0.0:1787828466.359652:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828466.359658:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828466.359661:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828466.359669:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972967552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828466.359683:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828466.359688:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828466.359690:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828466.359693:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4916 00000100:00080000:0.0:1787828466.359696:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828466.359699:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828466.359702:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828466.359710:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972967680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828466.359715:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828466.359724:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828466.359728:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828466.359729:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828466.359733:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4916 00000100:00080000:0.0:1787828466.359735:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828466.359738:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828466.359740:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828466.359748:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972967808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828466.359754:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828466.359756:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828466.359757:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828466.359760:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4916 00000100:00080000:0.0:1787828466.359763:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828466.359765:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff9c1a78860000 x1874668972967296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828466.359767:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828466.359773:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828466.359785:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972967936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828466.359791:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972967296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828466.359793:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828466.359801:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828466.359805:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828466.359812:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4916 00000100:00080000:0.0:1787828466.359816:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828466.359816:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828466.359822:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828466.359825:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828466.359826:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828466.359834:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972968064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828466.359836:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972967296:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828466.359874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972967424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828466.359896:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828466.359905:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff9c1a78863c00 x1874668972967424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828466.359916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972967424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.359925:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.359928:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828466.359932:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972967424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828466.359938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972967552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828466.359959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828466.359970:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff9c1a788630c0 x1874668972967552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828466.359979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972967552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.359984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.359986:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828466.359988:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972967552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828466.359992:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972967680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828466.360005:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828466.360012:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff9c1a78861680 x1874668972967680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828466.360019:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972967680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.360025:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.360027:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828466.360029:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972967680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828466.360032:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972967808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828466.360042:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828466.360048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff9c1a78862580 x1874668972967808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828466.360055:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972967808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.360060:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.360062:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828466.360064:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972967808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828466.360070:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972967936:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828466.360089:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828466.360100:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff9c1a78861e00 x1874668972967936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828466.360112:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972967936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.360141:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.360144:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828466.360146:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972967936:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828466.360151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972968064:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828466.360167:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828466.360175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828466/real 1787828466] req@ffff9c1a78860780 x1874668972968064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828466.360184:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972968064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828466.360190:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828466.360192:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828466.360196:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972968064:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828466.871506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828466.871507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828467.127625:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828467.895502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828467.895522:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828468.151638:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828468.919433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828468.919433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828469.175531:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828469.943486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828469.943486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828470.199547:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828470.967416:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828470.967419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828471.223579:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828471.479433:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828471.479443:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828471.479449:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828471.479460:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4921 00000100:00080000:0.0:1787828471.479472:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828471.479479:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828471.479537:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972968192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828471.479567:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828471.479571:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828471.479573:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828471.479578:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4921 00000100:00080000:0.0:1787828471.479582:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828471.479588:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828471.479599:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828471.479613:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972968320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828471.479622:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972968192:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828471.479630:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828471.479634:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828471.479636:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828471.479640:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4921 00000100:00080000:0.0:1787828471.479643:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828471.479647:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828471.479652:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828471.479664:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972968448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828471.479681:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828471.479685:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828471.479687:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828471.479691:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4921 00000100:00080000:0.0:1787828471.479694:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828471.479701:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828471.479702:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828471.479712:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828471.479725:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972968576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828471.479735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828471.479738:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828471.479741:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828471.479745:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4921 00000100:00080000:0.0:1787828471.479749:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828471.479753:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828471/real 1787828471] req@ffff9c1a788621c0 x1874668972968192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828471.479756:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828471.479765:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828471.479787:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972968704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828471.479789:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972968192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828471.479802:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828471.479812:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828471.479822:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972968192:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828471.479828:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828471.479834:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828471.479836:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828471.479841:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4921 00000100:00080000:0.0:1787828471.479845:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828471.479850:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828471.479856:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972968320:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787828471.479857:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828471.479880:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972968832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828471.479895:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828471.479899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828471.479907:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828471.479910:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828471.479913:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828471/real 1787828471] req@ffff9c1a78862940 x1874668972968320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828471.479923:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4921 00000100:00080000:0.0:1787828471.479936:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828471.479942:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972968320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828471.479950:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787828471.479965:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828471.479966:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828471.479974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828471.479978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972968320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828471.479989:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972968960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828471.479991:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972968448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828471.480033:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828471.480055:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828471/real 1787828471] req@ffff9c1a78861a40 x1874668972968448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828471.480076:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972968448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828471.480089:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828471.480094:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828471.480097:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972968448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828471.480104:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972968576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828471.480130:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828471.480145:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828471/real 1787828471] req@ffff9c1a78860f00 x1874668972968576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828471.480157:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972968576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828471.480166:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828471.480169:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828471.480172:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972968576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828471.480177:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972968704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828471.480191:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828471.480215:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828471/real 1787828471] req@ffff9c1a78862d00 x1874668972968704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828471.480226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972968704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828471.480234:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828471.480237:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828471.480240:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972968704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828471.480258:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972968832:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828471.480284:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828471.480294:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828471/real 1787828471] req@ffff9c1a78860b40 x1874668972968832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828471.480312:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972968832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828471.480320:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828471.480323:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828471.480326:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972968832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828471.480331:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972968960:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828471.480373:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828471.480384:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828471/real 1787828471] req@ffff9c1a78863840 x1874668972968960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828471.480395:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972968960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828471.480403:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828471.480406:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828471.480409:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972968960:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828471.991508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828471.991508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828472.247509:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828473.015460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828473.015509:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828473.271696:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828474.039503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828474.039503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828474.295782:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828475.063436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828475.063438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828475.319526:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828476.087486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828476.087486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828476.343568:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828476.599494:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828476.599508:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828476.599514:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828476.599528:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4926 00000100:00080000:0.0:1787828476.599553:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828476.599564:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828476.599617:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972969088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828476.599646:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828476.599650:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828476.599652:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828476.599655:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4926 00000100:00080000:0.0:1787828476.599658:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828476.599662:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828476.599673:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828476.599684:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972969216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828476.599691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828476.599694:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828476.599695:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828476.599698:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4926 00000100:00080000:0.0:1787828476.599699:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828476.599702:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828476.599704:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828476.599710:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972969344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828476.599725:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828476.599728:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828476.599729:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828476.599732:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4926 00000100:00080000:0.0:1787828476.599734:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828476.599737:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828476.599740:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828476.599747:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972969472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828476.599754:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828476.599757:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828476.599759:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828476.599762:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4926 00000100:00080000:0.0:1787828476.599763:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828476.599766:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828476.599770:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828476.599778:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972969600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828476.599786:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828476.599789:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828476.599791:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828476.599799:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4926 00000100:00100000:3.0:1787828476.599801:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972969088:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828476.599804:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828476.599816:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828476.599822:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828476.599837:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972969728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828476.599843:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828476.599846:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828476.599849:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828476.599852:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4926 00000100:00080000:0.0:1787828476.599856:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828476.599859:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828476.599863:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828476.599870:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972969856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828476.599890:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828476.599941:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828476/real 1787828476] req@ffff9c1a788612c0 x1874668972969088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828476.599960:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972969088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828476.599991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828476.599997:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828476.600003:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972969088:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828476.600013:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972969216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828476.600043:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828476.600058:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828476/real 1787828476] req@ffff9c1a78863840 x1874668972969216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828476.600073:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972969216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828476.600083:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828476.600087:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828476.600091:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972969216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828476.600098:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972969344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828476.600117:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828476.600127:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828476/real 1787828476] req@ffff9c1a78860b40 x1874668972969344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828476.600141:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972969344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828476.600151:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828476.600155:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828476.600158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972969344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828476.600166:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972969472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828476.600185:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828476.600196:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828476/real 1787828476] req@ffff9c1a78862d00 x1874668972969472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828476.600209:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972969472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828476.600220:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828476.600223:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828476.600227:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972969472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828476.600261:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972969600:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828476.600280:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828476.600292:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828476/real 1787828476] req@ffff9c1a78860f00 x1874668972969600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828476.600305:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972969600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828476.600316:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828476.600319:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828476.600323:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972969600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828476.600330:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972969728:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828476.600349:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828476.600399:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828476/real 1787828476] req@ffff9c1a78861a40 x1874668972969728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828476.600417:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972969728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828476.600427:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828476.600431:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828476.600435:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972969728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828476.600442:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972969856:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828476.600506:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828476.600518:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828476/real 1787828476] req@ffff9c1a78862940 x1874668972969856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828476.600529:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972969856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828476.600557:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828476.600561:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828476.600565:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972969856:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828477.111439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828477.111441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828477.367536:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828478.135415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828478.135416:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828478.391571:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828479.159483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828479.159521:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828479.415492:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828480.183495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828480.183495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828480.439552:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828481.207459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828481.207490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828481.463588:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828481.719489:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828481.719503:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828481.719507:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828481.719518:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4932 00000100:00080000:0.0:1787828481.719530:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828481.719539:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828481.719581:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972969984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828481.719615:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828481.719619:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828481.719622:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828481.719626:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4932 00000100:00080000:0.0:1787828481.719630:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828481.719634:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828481.719644:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828481.719656:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972970112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828481.719668:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828481.719671:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828481.719673:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828481.719676:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4932 00000100:00080000:0.0:1787828481.719679:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828481.719682:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828481.719686:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828481.719695:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972970240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828481.719707:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828481.719711:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828481.719713:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828481.719717:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4932 00000100:00080000:0.0:1787828481.719720:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828481.719723:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828481.719727:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828481.719736:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972970368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828481.719745:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972969984:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828481.719752:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828481.719757:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828481.719759:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828481.719763:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4932 00000100:00080000:0.0:1787828481.719767:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828481.719771:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828481.719775:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828481.719790:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972970496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828481.719800:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828481.719803:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828481.719805:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828481.719810:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4932 00000100:00080000:0.0:1787828481.719816:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828481.719821:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828481.719829:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828481.719835:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828481.719874:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972970624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828481.719896:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828481.719903:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828481.719911:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828481.719936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828481/real 1787828481] req@ffff9c1a788621c0 x1874668972969984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828481.719937:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4932 00000100:00080000:0.0:1787828481.719950:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828481.719962:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828481.719975:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828481.719986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972969984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828481.719997:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972970752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828481.720012:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828481.720034:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828481.720048:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972969984:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828481.720084:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972970112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828481.720165:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828481.720189:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828481/real 1787828481] req@ffff9c1a78860780 x1874668972970112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828481.720206:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972970112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828481.720232:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828481.720237:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828481.720241:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972970112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828481.720250:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972970240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828481.720295:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828481.720309:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828481/real 1787828481] req@ffff9c1a78861e00 x1874668972970240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828481.720323:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972970240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828481.720333:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828481.720336:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828481.720340:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972970240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828481.720440:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972970368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828481.720486:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828481.720500:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828481/real 1787828481] req@ffff9c1a78862580 x1874668972970368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828481.720513:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972970368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828481.720522:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828481.720525:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828481.720529:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972970368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828481.720535:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972970496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828481.720551:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828481.720560:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828481/real 1787828481] req@ffff9c1a78861680 x1874668972970496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828481.720572:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972970496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828481.720580:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828481.720583:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828481.720585:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972970496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828481.720591:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972970624:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828481.720687:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828481.720703:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828481/real 1787828481] req@ffff9c1a788630c0 x1874668972970624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828481.720716:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972970624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828481.720725:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828481.720728:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828481.720731:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972970624:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828481.720737:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972970752:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828481.720759:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828481.720768:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828481/real 1787828481] req@ffff9c1a78863c00 x1874668972970752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828481.720779:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972970752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828481.720787:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828481.720791:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828481.720794:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972970752:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828482.231475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828482.231491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828482.487579:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828483.255470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828483.255485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828483.511579:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828484.279437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828484.279437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828484.535609:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828485.303517:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828485.303519:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828485.559551:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828486.327506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828486.327512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828486.583523:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828486.839563:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828486.839575:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828486.839581:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828486.839594:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4937 00000100:00080000:3.0:1787828486.839607:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828486.839616:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828486.839676:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972970880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828486.839728:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828486.839735:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828486.839739:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828486.839747:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4937 00000100:00080000:3.0:1787828486.839754:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828486.839763:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828486.839779:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828486.839798:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972970880:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828486.839805:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972971008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828486.839835:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828486.839841:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828486.839844:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828486.839850:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4937 00000100:00080000:3.0:1787828486.839856:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828486.839862:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828486.839871:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828486.839886:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972971136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828486.839902:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828486.839905:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828486.839910:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828486.839913:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828486.839920:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4937 00000100:00080000:3.0:1787828486.839926:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828486.839932:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828486.839940:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828486.839957:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972971264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828486.839962:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828486/real 1787828486] req@ffff9c1a7895e580 x1874668972970880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828486.839976:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828486.839982:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828486.839985:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828486.839992:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4937 00000100:00100000:0.0:1787828486.839994:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972970880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828486.839998:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828486.840005:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828486.840013:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828486.840014:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828486.840022:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828486.840030:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972970880:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828486.840030:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972971392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828486.840046:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828486.840051:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828486.840054:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828486.840060:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4937 00000100:00100000:0.0:1787828486.840067:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972971008:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828486.840067:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828486.840074:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828486.840083:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828486.840099:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972971520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828486.840106:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828486.840119:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828486.840124:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828486.840128:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828486.840129:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828486/real 1787828486] req@ffff9c1a7895c000 x1874668972971008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828486.840136:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4937 00000100:00080000:3.0:1787828486.840143:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828486.840150:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828486.840153:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972971008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828486.840159:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828486.840170:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828486.840176:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972971648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828486.840177:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828486.840185:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972971008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828486.840196:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972971136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828486.840234:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828486.840257:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828486/real 1787828486] req@ffff9c1a7895d2c0 x1874668972971136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828486.840283:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972971136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828486.840296:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828486.840301:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828486.840305:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972971136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828486.840313:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972971264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828486.840400:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828486.840421:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828486/real 1787828486] req@ffff9c1a7895f480 x1874668972971264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828486.840436:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972971264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828486.840447:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828486.840451:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828486.840455:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972971264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828486.840463:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972971392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828486.840489:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828486.840506:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828486/real 1787828486] req@ffff9c1a7895cf00 x1874668972971392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828486.840543:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972971392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828486.840560:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828486.840567:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828486.840573:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972971392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828486.840611:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972971520:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828486.840664:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828486.840688:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828486/real 1787828486] req@ffff9c1a7895c3c0 x1874668972971520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828486.840714:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972971520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828486.840731:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828486.840738:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828486.840744:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972971520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828486.840756:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972971648:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828486.840788:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828486.840802:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828486/real 1787828486] req@ffff9c1a7895e940 x1874668972971648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828486.840826:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972971648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828486.840845:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828486.840849:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828486.840853:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972971648:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828487.351471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828487.351479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828487.607585:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828488.375505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828488.375511:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828488.631581:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828489.399512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828489.399513:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828489.655663:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828490.423453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828490.423476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828490.679495:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828491.447448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828491.447448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828491.703527:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828491.959440:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828491.959449:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828491.959454:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828491.959464:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4942 00000100:00080000:3.0:1787828491.959475:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828491.959483:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828491.959517:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972971776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828491.959563:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828491.959568:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828491.959572:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828491.959577:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4942 00000100:00080000:3.0:1787828491.959583:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828491.959588:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828491.959610:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828491.959630:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972971904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828491.959642:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972971776:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828491.959646:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828491.959655:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828491.959657:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828491.959662:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4942 00000100:00080000:3.0:1787828491.959666:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828491.959670:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828491.959675:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828491.959688:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972972032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828491.959698:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828491.959701:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828491.959702:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828491.959705:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4942 00000100:00080000:3.0:1787828491.959712:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828491.959716:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828491.959720:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828491.959725:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828491.959732:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972972160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828491.959773:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828491/real 1787828491] req@ffff9c1a7895fc00 x1874668972971776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828491.959796:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972971776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828491.959822:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828491.959828:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828491.959834:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828491.959837:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972971776:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828491.959838:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828491.959842:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828491.959849:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4942 00000100:00080000:3.0:1787828491.959862:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828491.959891:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828491.959897:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828491.959898:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972971904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828491.959914:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972972288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828491.959925:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828491.959928:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828491.959930:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828491.959934:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4942 00000800:00000100:1.0:1787828491.959941:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828491.959941:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828491.959950:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828491.959963:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828491.959974:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828491/real 1787828491] req@ffff9c1a7895de00 x1874668972971904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828491.959986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972971904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828491.959999:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828491.960002:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828491.960005:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972971904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828491.960010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972972032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828491.960035:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828491.960053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828491/real 1787828491] req@ffff9c1a7895c780 x1874668972972032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828491.960071:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972972032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828491.960098:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828491.960106:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828491.960121:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972972032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828491.960123:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972972416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828491.960130:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972972160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828491.960153:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828491.960163:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828491/real 1787828491] req@ffff9c1a7895f840 x1874668972972160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828491.960171:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972972160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828491.960188:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828491.960209:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828491.960216:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972972160:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828491.960227:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828491.960231:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828491.960234:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828491.960239:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4942 00000100:00080000:3.0:1787828491.960245:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828491.960247:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972972288:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828491.960253:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828491.960260:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828491.960272:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828491.960276:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972972544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828491.960285:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828491/real 1787828491] req@ffff9c1a7895da40 x1874668972972288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828491.960300:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972972288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828491.960309:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828491.960312:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828491.960315:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972972288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828491.960320:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972972416:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828491.960334:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828491.960398:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828491/real 1787828491] req@ffff9c1a7895cb40 x1874668972972416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828491.960411:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972972416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828491.960418:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828491.960421:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828491.960424:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972972416:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828491.960436:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972972544:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828491.960454:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828491.960462:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828491/real 1787828491] req@ffff9c1a7895f0c0 x1874668972972544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828491.960471:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972972544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828491.960477:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828491.960480:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828491.960482:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972972544:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828492.471466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828492.471507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828492.727629:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828493.495482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828493.495482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828493.751514:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828494.519454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828494.519573:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828494.775621:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828495.543467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828495.543477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828495.799586:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828496.567480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828496.567498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828496.823655:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828497.079481:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828497.079496:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828497.079502:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.079515:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4947 00000100:00080000:0.0:1787828497.079532:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828497.079574:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828497.079647:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972972672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828497.079691:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828497.079696:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828497.079699:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.079705:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4947 00000100:00080000:0.0:1787828497.079711:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828497.079717:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828497.079747:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828497.079799:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972972672:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828497.079807:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972972800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828497.079843:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828497.079850:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828497.079854:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.079861:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4947 00000100:00080000:0.0:1787828497.079869:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828497.079877:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828497.079900:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828497.079931:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972972928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828497.079945:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828497.079960:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828497.079965:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828497.079969:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.079977:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4947 00000100:00080000:0.0:1787828497.079985:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828497.079994:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828497.080003:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828497.080016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff9c1a78860000 x1874668972972672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828497.080031:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972973056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828497.080049:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972972672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828497.080073:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828497.080080:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828497.080081:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828497.080083:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.080097:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4947 00000100:00080000:0.0:1787828497.080105:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828497.080113:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828497.080116:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828497.080140:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972972672:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787828497.080141:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828497.080174:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972973184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828497.080191:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828497.080191:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972972800:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828497.080203:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828497.080209:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828497.080248:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4947 00000100:00080000:0.0:1787828497.080254:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828497.080260:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828497.080271:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828497.080293:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828497.080302:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972973312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828497.080416:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff9c1a78863c00 x1874668972972800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828497.080450:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828497.080456:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828497.080459:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828497.080468:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972972800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828497.080470:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4947 00000100:00080000:0.0:1787828497.080481:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828497.080496:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828497.080501:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828497.080506:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828497.080514:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828497.080523:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972972800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828497.080531:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972973440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828497.080540:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972972928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828497.080616:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828497.080639:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff9c1a788630c0 x1874668972972928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828497.080655:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972972928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828497.080666:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828497.080670:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828497.080674:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972972928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828497.080681:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972973056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828497.080703:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828497.080722:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff9c1a78861680 x1874668972973056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828497.080761:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972973056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828497.080789:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828497.080795:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828497.080802:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972973056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828497.080836:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972973184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828497.080879:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828497.080903:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff9c1a78862580 x1874668972973184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828497.080926:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972973184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828497.080937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828497.080941:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828497.080946:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972973184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828497.080954:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972973312:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828497.080981:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828497.080994:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff9c1a78861e00 x1874668972973312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828497.081008:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972973312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828497.081028:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828497.081032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828497.081035:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972973312:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828497.081042:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972973440:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828497.081070:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828497.081089:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828497/real 1787828497] req@ffff9c1a78860780 x1874668972973440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828497.081113:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972973440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828497.081131:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828497.081167:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828497.081173:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972973440:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828497.591518:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828497.591520:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828497.847639:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828498.615488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828498.615492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828498.871592:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828499.639448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828499.639472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828499.895556:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828500.663478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828500.663542:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828500.919594:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828501.687490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828501.687537:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828501.943567:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828502.199525:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828502.199545:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828502.199552:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828502.199568:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4952 00000100:00080000:2.0:1787828502.199586:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828502.199597:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828502.199654:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668972973568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828502.199694:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828502.199701:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828502.199705:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828502.199714:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4952 00000100:00080000:2.0:1787828502.199722:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828502.199730:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828502.199749:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828502.199777:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972973568:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828502.199786:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668972973696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828502.199800:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828502.199804:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828502.199807:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828502.199811:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4952 00000100:00080000:2.0:1787828502.199816:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828502.199822:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828502.199843:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828502.199852:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668972973824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828502.199864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828502.199868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828502.199870:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828502.199874:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4952 00000100:00080000:2.0:1787828502.199877:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828502.199882:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828502.199887:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828502.199899:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668972973952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828502.199909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828502.199913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828502.199916:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828502.199920:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4952 00000100:00080000:2.0:1787828502.199924:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828502.199928:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972973696:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828502.199928:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828502.199935:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828502.199947:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668972974080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828502.199959:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828502.199963:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828502.199965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972973824:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828502.199965:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828502.199971:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4952 00000100:00100000:1.0:1787828502.199990:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972973952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828502.200016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972974080:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828502.201776:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828502.201789:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828502.201803:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828502.201838:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668972974208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828502.201866:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828502.201871:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828502.201874:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828502.201880:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4952 00000100:00080000:2.0:1787828502.201884:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828502.201888:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828502.201893:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828502.201908:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668972974336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828502.201926:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972974208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828502.202003:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972974336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828502.711491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828502.711492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828502.967630:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828503.735422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828503.735491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828503.991671:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828504.759513:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828504.759554:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828505.015566:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828505.272008:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787828505.272105:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828505.272133:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828505.272197:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828505.272212:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828505.272226:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828505.272242:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828505.272277:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787828505.272280:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828505] req@ffff9c1a49d8cb40 x1874668972973696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787828505.272293:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787828505.272397:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668972973696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828505.272420:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828505.272430:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828505.272438:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972973696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828505.272451:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828505] req@ffff9c1a49d8da40 x1874668972973824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828505.272480:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668972973824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828505.272493:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828505.272498:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828505.272503:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972973824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828505.272513:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828505] req@ffff9c1a49d8cf00 x1874668972973952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828505.272529:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668972973952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828505.272541:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828505.272546:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828505.272550:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972973952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828505.272561:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828505] req@ffff9c1a49d8c000 x1874668972973568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828505.272580:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668972973568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828505.272591:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828505.272597:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828505.272602:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972973568:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828505.272644:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828505] req@ffff9c1a49d8c780 x1874668972974336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828505.272665:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668972974336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828505.272678:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828505.272684:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828505.272689:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972974336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828505.272700:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828505] req@ffff9c1a49d8de00 x1874668972974208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828505.272716:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668972974208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828505.272729:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828505.272734:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828505.272739:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972974208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828505.272750:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828502/real 1787828505] req@ffff9c1a49d8f0c0 x1874668972974080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828505.272770:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668972974080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828505.272782:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828505.272787:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828505.272792:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972974080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828505.783465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828505.783505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828506.039607:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828506.807482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828506.807482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828507.063532:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828507.319487:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828507.319495:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828507.319500:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828507.319509:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4957 00000100:00080000:3.0:1787828507.319519:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828507.319527:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828507.319565:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972974464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828507.319599:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828507.319603:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828507.319606:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828507.319612:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4957 00000100:00080000:3.0:1787828507.319618:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828507.319624:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828507.319634:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828507.319644:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972974464:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828507.319656:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972974592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828507.319674:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828507.319678:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828507.319686:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828507.319691:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4957 00000100:00080000:3.0:1787828507.319696:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828507.319702:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828507.319709:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828507.319726:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972974720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828507.319731:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828507.319753:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828507.319758:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828507.319761:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828507.319768:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4957 00000100:00080000:3.0:1787828507.319772:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828507.319777:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828507.319784:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828507.319786:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828507] req@ffff9c1a7895e1c0 x1874668972974464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828507.319805:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972974848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828507.319818:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972974464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828507.319826:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828507.319838:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828507.319840:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828507.319842:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828507.319849:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4957 00000100:00080000:0.0:1787828507.319854:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828507.319865:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972974464:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828507.319869:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828507.319878:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828507.319884:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828507.319901:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972974976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828507.319911:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972974592:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828507.319935:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828507.319940:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828507.319943:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828507.319953:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4957 00000800:00000100:0.0:1787828507.319962:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828507.319963:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828507.319974:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828507.319984:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828507.319992:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828507] req@ffff9c1a7895f0c0 x1874668972974592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828507.320009:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972975104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828507.320026:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972974592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828507.320028:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828507.320049:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828507.320055:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828507.320060:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828507.320066:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4957 00000100:00080000:0.0:1787828507.320072:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828507.320079:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828507.320083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972974592:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828507.320093:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828507.320101:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828507.320103:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972974720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828507.320119:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972975232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828507.320138:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828507.320158:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828507] req@ffff9c1a7895cb40 x1874668972974720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828507.320174:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972974720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828507.320190:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828507.320194:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828507.320198:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972974720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828507.320206:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972974848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828507.320228:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828507.320270:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828507] req@ffff9c1a7895da40 x1874668972974848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828507.320282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972974848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828507.320297:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828507.320302:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828507.320305:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972974848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828507.320323:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972974976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828507.320382:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828507.320398:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828507] req@ffff9c1a7895f840 x1874668972974976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828507.320410:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972974976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828507.320418:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828507.320422:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828507.320425:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972974976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828507.320433:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972975104:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828507.320456:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828507.320474:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828507] req@ffff9c1a7895c780 x1874668972975104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828507.320487:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972975104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828507.320495:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828507.320498:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828507.320503:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972975104:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828507.320511:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972975232:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828507.320533:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828507.320545:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828507/real 1787828507] req@ffff9c1a7895de00 x1874668972975232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828507.320558:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972975232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828507.320566:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828507.320569:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828507.320572:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972975232:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828507.831431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828507.831432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828508.087515:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828508.855486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828508.855496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828509.111620:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828509.879430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828509.879523:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828510.135764:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828510.903443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828510.903551:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828511.159635:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828511.927453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828511.927476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828512.183646:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828512.439517:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828512.439529:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828512.439535:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828512.439548:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4962 00000100:00080000:0.0:1787828512.439573:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828512.439583:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828512.439639:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972975360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828512.439740:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828512.439747:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828512.439751:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828512.439758:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4962 00000100:00080000:0.0:1787828512.439766:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828512.439773:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828512.439786:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828512.439807:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972975488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828512.439823:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972975360:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828512.439830:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828512.439841:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828512.439843:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828512.439849:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4962 00000100:00080000:0.0:1787828512.439852:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828512.439856:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828512.439863:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828512.439877:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972975616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828512.439892:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828512.439896:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828512.439899:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828512.439903:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4962 00000100:00080000:0.0:1787828512.439908:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828512.439912:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828512.439917:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828512.439935:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828512.439939:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972975744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828512.439983:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828512.439989:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828512.439993:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828512.440002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff9c1a788621c0 x1874668972975360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828512.440005:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4962 00000100:00080000:0.0:1787828512.440020:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828512.440034:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828512.440043:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972975360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828512.440049:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828512.440081:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828512.440087:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828512.440094:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972975360:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828512.440102:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972975872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828512.440126:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828512.440134:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972975488:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828512.440135:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828512.440143:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828512.440153:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4962 00000100:00080000:0.0:1787828512.440160:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828512.440168:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828512.440177:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828512.440179:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828512.440234:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972976000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828512.440235:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff9c1a78862940 x1874668972975488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828512.440288:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828512.440297:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828512.440300:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828512.440304:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972975488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828512.440306:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4962 00000100:00080000:0.0:1787828512.440319:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828512.440330:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828512.440332:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828512.440338:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828512.440383:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828512.440392:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972975488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828512.440413:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972976128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828512.440419:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972975616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828512.440469:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828512.440486:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff9c1a78861a40 x1874668972975616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828512.440501:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972975616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828512.440517:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828512.440521:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828512.440525:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972975616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828512.440532:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972975744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828512.440552:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828512.440563:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff9c1a78860f00 x1874668972975744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828512.440586:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972975744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828512.440608:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828512.440612:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828512.440616:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972975744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828512.440623:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972975872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828512.440644:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828512.440655:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff9c1a78862d00 x1874668972975872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828512.440675:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972975872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828512.440692:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828512.440699:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828512.440705:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972975872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828512.440751:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972976000:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828512.440794:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828512.440826:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff9c1a78860b40 x1874668972976000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828512.440852:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972976000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828512.440868:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828512.440874:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828512.440881:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972976000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828512.440893:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972976128:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828512.440927:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828512.440947:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828512/real 1787828512] req@ffff9c1a78863840 x1874668972976128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828512.440981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972976128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828512.440996:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828512.441002:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828512.441008:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972976128:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828512.951457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828512.951555:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828513.207613:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828513.975448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828513.975463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828514.231583:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828514.999459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828514.999486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828515.255529:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828516.023488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828516.023488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828516.279645:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828517.047518:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828517.047528:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828517.303611:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828517.559426:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828517.559437:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828517.559442:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828517.559453:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4967 00000100:00080000:0.0:1787828517.559464:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828517.559473:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828517.559512:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972976256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828517.559544:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828517.559548:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828517.559550:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828517.559554:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4967 00000100:00080000:0.0:1787828517.559558:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828517.559562:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828517.559572:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828517.559586:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972976384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828517.559598:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828517.559604:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828517.559609:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828517.559609:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972976256:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828517.559623:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4967 00000100:00080000:0.0:1787828517.559631:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828517.559637:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828517.559646:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828517.559668:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972976512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828517.559687:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828517.559695:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787828517.559696:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828517.559700:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828517.559731:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4967 00000100:00080000:0.0:1787828517.559737:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828517.559743:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828517.559749:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828517.559758:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828517/real 1787828517] req@ffff9c1a788612c0 x1874668972976256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828517.559768:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972976640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828517.559784:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828517.559789:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972976256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828517.559790:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828517.559795:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828517.559806:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4967 00000100:00080000:0.0:1787828517.559824:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828517.559825:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828517.559831:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828517.559836:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787828517.559841:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828517.559843:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972976256:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828517.559858:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972976768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828517.559872:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828517.559876:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972976384:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828517.559883:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828517.559898:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828517.559907:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4967 00000100:00080000:0.0:1787828517.559916:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828517.559923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828517.559929:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828517.559940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828517.559943:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828517/real 1787828517] req@ffff9c1a78863840 x1874668972976384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828517.559972:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972976896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828517.559973:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972976384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828517.560020:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828517.560021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828517.560027:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828517.560030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828517.560037:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828517.560037:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972976384:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828517.560048:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4967 00000100:00100000:1.0:1787828517.560054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972976512:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828517.560056:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828517.560067:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828517.560073:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828517.560086:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828517.560092:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972977024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828517.560105:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828517/real 1787828517] req@ffff9c1a78860b40 x1874668972976512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828517.560125:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972976512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828517.560145:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828517.560148:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828517.560151:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972976512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828517.560182:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972976640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828517.560226:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828517.560254:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828517/real 1787828517] req@ffff9c1a78862d00 x1874668972976640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828517.560276:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972976640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828517.560289:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828517.560294:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828517.560299:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972976640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828517.560320:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972976768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828517.560384:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828517.560397:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828517/real 1787828517] req@ffff9c1a78860f00 x1874668972976768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828517.560409:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972976768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828517.560436:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828517.560439:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828517.560442:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972976768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828517.560448:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972976896:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828517.560467:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828517.560476:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828517/real 1787828517] req@ffff9c1a78861a40 x1874668972976896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828517.560487:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972976896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828517.560495:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828517.560498:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828517.560500:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972976896:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828517.560506:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972977024:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828517.560526:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828517.560534:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828517/real 1787828517] req@ffff9c1a78862940 x1874668972977024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828517.560545:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972977024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828517.560552:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828517.560554:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828517.560558:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972977024:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828518.071487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828518.071489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828518.327646:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828519.095453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828519.095460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828519.351604:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828520.119504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828520.119504:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828520.375546:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828521.143433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828521.143469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828521.399552:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828522.167487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828522.167487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828522.423495:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828522.679462:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828522.679473:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828522.679478:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828522.679490:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4973 00000100:00080000:3.0:1787828522.679502:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828522.679509:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828522.679560:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972977152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828522.679589:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828522.679593:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828522.679595:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828522.679600:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4973 00000100:00080000:3.0:1787828522.679603:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828522.679607:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828522.679618:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828522.679632:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972977280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828522.679644:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972977152:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828522.679647:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828522.679657:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828522.679660:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828522.679664:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4973 00000100:00080000:3.0:1787828522.679668:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828522.679671:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828522.679676:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828522.679689:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972977408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828522.679700:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828522.679704:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828522.679705:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828522.679709:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4973 00000100:00080000:3.0:1787828522.679712:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828522.679716:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828522.679720:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828522.679731:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972977536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828522.679739:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828522.679742:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828522.679743:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828522.679749:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4973 00000100:00080000:3.0:1787828522.679756:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828522.679758:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828522.679763:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828522.679770:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828522.679780:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972977664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828522.679790:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828522.679793:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828522.679795:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828522.679798:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4973 00000100:00080000:3.0:1787828522.679802:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828522.679805:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828522.679810:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828522.679826:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828522/real 1787828522] req@ffff9c1a7895fc00 x1874668972977152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828522.679827:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972977792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828522.679883:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828522.679891:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828522.679893:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828522.679900:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4973 00000100:00080000:3.0:1787828522.679905:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828522.679910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972977152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828522.679928:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828522.679929:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828522.679939:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828522.679942:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828522.679951:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972977152:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828522.679960:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972977920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828522.679987:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972977280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828522.680023:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828522.680038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828522/real 1787828522] req@ffff9c1a7895e940 x1874668972977280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828522.680053:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972977280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828522.680064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828522.680068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828522.680072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972977280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828522.680080:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972977408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828522.680100:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828522.680110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828522/real 1787828522] req@ffff9c1a7895c3c0 x1874668972977408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828522.680124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972977408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828522.680165:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828522.680170:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828522.680174:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972977408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828522.680182:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972977536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828522.680209:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828522.680223:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828522/real 1787828522] req@ffff9c1a7895cf00 x1874668972977536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828522.680237:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972977536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828522.680247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828522.680251:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828522.680255:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972977536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828522.680260:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972977664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828522.680280:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828522.680290:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828522/real 1787828522] req@ffff9c1a7895f480 x1874668972977664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828522.680303:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972977664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828522.680313:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828522.680316:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828522.680319:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972977664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828522.680325:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972977792:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828522.680380:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828522.680395:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828522/real 1787828522] req@ffff9c1a7895d2c0 x1874668972977792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828522.680410:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972977792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828522.680419:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828522.680423:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828522.680427:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972977792:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828522.680434:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972977920:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828522.680457:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828522.680468:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828522/real 1787828522] req@ffff9c1a7895c000 x1874668972977920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828522.680500:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972977920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828522.680515:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828522.680520:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828522.680526:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972977920:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828523.191466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828523.191501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828523.447617:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828524.215462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828524.215489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828524.471563:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828525.239436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828525.239449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828525.495628:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828526.263502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828526.263505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828526.519597:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828527.287510:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828527.287510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828527.543561:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828527.799504:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828527.799550:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828527.799558:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828527.799574:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4978 00000100:00080000:3.0:1787828527.799591:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828527.799603:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828527.799702:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972978048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828527.799760:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828527.799768:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828527.799772:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828527.799782:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4978 00000100:00080000:3.0:1787828527.799791:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828527.799801:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828527.799834:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828527.799868:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972978048:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828527.799870:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972978176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828527.799892:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828527.799902:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828527.799908:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828527.799913:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4978 00000100:00080000:3.0:1787828527.799921:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828527.799928:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828527.799936:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828527.800026:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828527.800111:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972978304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828527.800133:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828527.800141:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787828527.800145:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828527/real 1787828527] req@ffff9c1a7895e580 x1874668972978048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828527.800146:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828527.800159:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4978 00000100:00080000:3.0:1787828527.800170:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828527.800180:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828527.800190:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828527.800203:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972978048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828527.800229:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828527.800238:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828527.800262:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972978048:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828527.800310:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972978176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828527.800460:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828527.800476:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972978432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828527.800498:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828527.800507:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828527/real 1787828527] req@ffff9c1a7895c000 x1874668972978176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828527.800507:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828527.800515:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828527.800568:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972978176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828527.800589:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828527.800600:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828527.800609:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4978 00000100:00100000:0.0:1787828527.800615:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972978176:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828527.800621:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828527.800634:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828527.800641:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972978304:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787828527.800647:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828527.800673:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972978560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828527.800698:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828527.800716:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828527.800726:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828527.800728:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828527/real 1787828527] req@ffff9c1a7895d2c0 x1874668972978304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828527.800733:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828527.800753:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4978 00000100:00080000:3.0:1787828527.800767:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828527.800773:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972978304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828527.800782:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828527.800804:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828527.800807:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828527.800821:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828527.800829:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972978304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828527.800837:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972978688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828527.800853:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828527.800858:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828527.800861:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828527.800869:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4978 00000100:00100000:0.0:1787828527.800871:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972978432:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828527.800875:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828527.800888:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828527.800898:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828527.800915:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972978816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828527.800936:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828527.800983:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828527/real 1787828527] req@ffff9c1a7895f480 x1874668972978432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828527.801010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972978432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828527.801029:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828527.801036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828527.801043:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972978432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828527.801056:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972978560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828527.801122:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828527.801161:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828527/real 1787828527] req@ffff9c1a7895cf00 x1874668972978560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828527.801184:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972978560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828527.801213:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828527.801219:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828527.801226:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972978560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828527.801239:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972978688:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828527.801278:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828527.801316:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828527/real 1787828527] req@ffff9c1a7895c3c0 x1874668972978688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828527.801341:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972978688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828527.801358:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828527.801365:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828527.801371:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972978688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828527.801465:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972978816:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828527.801514:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828527.801552:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828527/real 1787828527] req@ffff9c1a7895e940 x1874668972978816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828527.801575:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972978816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828527.801585:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828527.801589:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828527.801594:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972978816:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828528.311507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828528.311507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828528.567627:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828529.335526:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828529.335555:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828529.591530:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828530.359514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828530.359524:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828530.615614:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828531.383477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828531.383541:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828531.639656:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828532.407501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828532.407572:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828532.663562:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828532.919464:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828532.919474:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828532.919480:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828532.919491:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4983 00000100:00080000:3.0:1787828532.919502:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828532.919509:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828532.919552:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972978944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828532.919585:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828532.919589:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828532.919591:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828532.919595:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4983 00000100:00080000:3.0:1787828532.919598:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828532.919602:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828532.919611:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828532.919622:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972979072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828532.919631:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828532.919635:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828532.919637:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828532.919640:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4983 00000100:00080000:3.0:1787828532.919642:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828532.919645:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828532.919649:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828532.919660:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972979200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828532.919680:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828532.919685:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828532.919689:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828532.919691:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972978944:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828532.919696:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4983 00000100:00080000:3.0:1787828532.919714:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828532.919719:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828532.919724:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828532.919738:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972979328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828532.919748:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828532.919752:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828532.919754:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828532.919757:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4983 00000100:00080000:3.0:1787828532.919761:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828532.919764:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828532.919768:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828532.919774:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828532.919782:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972979456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828532.919792:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828532.919795:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828532.919797:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828532.919801:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4983 00000100:00080000:3.0:1787828532.919806:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828532.919811:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828532.919816:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828532.919828:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972979584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828532.919839:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828532.919840:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828532/real 1787828532] req@ffff9c1a7895fc00 x1874668972978944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828532.919845:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828532.919851:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828532.919863:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4983 00000100:00080000:3.0:1787828532.919870:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828532.919874:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828532.919879:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828532.919881:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972978944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828532.919894:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972979712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828532.919897:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828532.919904:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828532.919923:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972978944:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828532.919932:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972979072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828532.919959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828532.919972:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828532/real 1787828532] req@ffff9c1a7895de00 x1874668972979072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828532.919984:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972979072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828532.919992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828532.919995:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828532.919998:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972979072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828532.920022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972979200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828532.920041:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828532.920051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828532/real 1787828532] req@ffff9c1a7895c780 x1874668972979200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828532.920090:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972979200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828532.920098:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828532.920101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828532.920104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972979200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828532.920110:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972979328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828532.920131:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828532.920142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828532/real 1787828532] req@ffff9c1a7895f840 x1874668972979328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828532.920152:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972979328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828532.920161:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828532.920164:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828532.920166:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972979328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828532.920171:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972979456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828532.920185:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828532.920194:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828532/real 1787828532] req@ffff9c1a7895da40 x1874668972979456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828532.920211:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972979456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828532.920218:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828532.920220:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828532.920223:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972979456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828532.920228:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972979584:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828532.920243:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828532.920251:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828532/real 1787828532] req@ffff9c1a7895cb40 x1874668972979584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828532.920261:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972979584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828532.920269:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828532.920271:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828532.920274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972979584:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828532.920278:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972979712:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828532.920293:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828532.920301:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828532/real 1787828532] req@ffff9c1a7895f0c0 x1874668972979712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828532.920311:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972979712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828532.920325:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828532.920328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828532.920331:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972979712:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828533.431463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828533.431493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828533.687599:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828534.455492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828534.455607:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828534.711603:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828535.479421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828535.479424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828535.735540:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828536.503482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828536.503483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828536.759531:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828537.527439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828537.527440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828537.783609:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828538.039458:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828538.039483:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828538.039489:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828538.039502:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4988 00000100:00080000:3.0:1787828538.039516:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828538.039528:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828538.039586:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972979840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828538.039637:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828538.039645:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828538.039648:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828538.039655:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4988 00000100:00080000:3.0:1787828538.039661:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828538.039668:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828538.039682:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828538.039723:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972979968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828538.039742:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828538.039748:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828538.039752:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828538.039767:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972979840:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828538.039784:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4988 00000100:00080000:3.0:1787828538.039793:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828538.039803:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828538.039816:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828538.039842:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972980096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828538.039856:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828538.039861:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828538.039866:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828538.039872:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4988 00000100:00080000:3.0:1787828538.039880:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828538.039896:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828538.039908:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828538.039919:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828538.039938:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972980224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828538.039953:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828538.039958:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828538.039960:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828538.039972:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4988 00000100:00100000:0.0:1787828538.039975:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff9c1a7895e1c0 x1874668972979840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828538.039980:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828538.039996:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828538.040060:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972979840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828538.040063:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828538.040096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828538.040102:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972980352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828538.040110:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828538.040134:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972979840:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828538.040149:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828538.040154:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828538.040158:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828538.040165:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4988 00000100:00080000:3.0:1787828538.040172:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828538.040177:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828538.040184:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828538.040197:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972979968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828538.040199:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972980480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828538.040241:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828538.040245:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828538.040248:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828538.040256:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4988 00000100:00080000:3.0:1787828538.040260:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828538.040267:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828538.040278:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828538.040291:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828538.040301:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff9c1a7895f0c0 x1874668972979968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828538.040320:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972980608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828538.040321:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972979968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828538.040467:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828538.040472:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828538.040476:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972979968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828538.040484:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972980096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828538.040512:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828538.040531:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff9c1a7895cb40 x1874668972980096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828538.040544:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972980096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828538.040552:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828538.040555:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828538.040558:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972980096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828538.040564:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972980224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828538.040580:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828538.040589:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff9c1a7895da40 x1874668972980224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828538.040599:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972980224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828538.040607:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828538.040610:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828538.040613:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972980224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828538.040621:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972980352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828538.040647:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828538.040663:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff9c1a7895f840 x1874668972980352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828538.040674:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972980352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828538.040681:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828538.040684:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828538.040686:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972980352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828538.040718:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972980480:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828538.040744:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828538.040755:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff9c1a7895c780 x1874668972980480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828538.040765:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972980480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828538.040772:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828538.040774:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828538.040777:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972980480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828538.040783:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972980608:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828538.040796:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828538.040805:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828538/real 1787828538] req@ffff9c1a7895de00 x1874668972980608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828538.040822:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972980608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828538.040829:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828538.040831:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828538.040834:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972980608:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828538.551417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828538.551520:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828538.807634:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828539.575432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828539.575432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828539.831546:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828540.599474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828540.599532:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828540.855649:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828541.623506:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828541.623507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828541.879655:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828542.647433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828542.647448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828542.903575:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828543.159544:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.159562:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828543.159569:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.159585:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4993 00000100:00080000:0.0:1787828543.159600:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.159610:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828543.159657:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972980736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828543.159721:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.159725:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828543.159727:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.159731:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4993 00000100:00080000:0.0:1787828543.159734:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.159738:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828543.159758:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828543.159773:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972980864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828543.159793:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.159797:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828543.159797:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972980736:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828543.159801:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.159820:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4993 00000100:00080000:0.0:1787828543.159824:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.159827:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828543.159831:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828543.159842:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972980992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828543.159852:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.159855:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828543.159857:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.159860:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4993 00000100:00080000:0.0:1787828543.159886:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.159890:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828543.159913:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828543.159929:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828543.159933:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972981120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828543.159949:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.159954:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828543.159956:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.159963:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4993 00000100:00080000:0.0:1787828543.159969:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.159974:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828543.159980:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828543.160024:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972981248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828543.160038:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.160046:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828543.160048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff9c1a788621c0 x1874668972980736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828543.160049:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828543.160077:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4993 00000100:00080000:0.0:1787828543.160088:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.160095:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828543.160104:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828543.160105:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972980736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828543.160127:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972981376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828543.160130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.160145:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828543.160158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972980736:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828543.160178:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828543.160187:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828543.160192:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828543.160201:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972980864:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828543.160204:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4993 00000100:00080000:0.0:1787828543.160221:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828543.160232:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828543.160243:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828543.160254:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828543.160288:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972981504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828543.160288:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff9c1a78860780 x1874668972980864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828543.160339:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972980864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.160395:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.160400:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828543.160405:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972980864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828543.160414:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972980992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828543.160461:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828543.160477:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff9c1a78861e00 x1874668972980992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828543.160493:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972980992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.160504:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.160508:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828543.160512:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972980992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828543.160520:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972981120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828543.160540:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828543.160551:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff9c1a78862580 x1874668972981120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828543.160565:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972981120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.160576:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.160579:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828543.160583:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972981120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828543.160589:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972981248:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828543.160607:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828543.160618:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff9c1a78861680 x1874668972981248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828543.160641:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972981248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.160651:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.160654:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828543.160658:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972981248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828543.160664:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972981376:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828543.160683:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828543.160709:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff9c1a788630c0 x1874668972981376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828543.160719:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972981376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.160725:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.160727:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828543.160729:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972981376:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828543.160744:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972981504:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828543.160759:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828543.160794:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828543/real 1787828543] req@ffff9c1a78863c00 x1874668972981504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828543.160818:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972981504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828543.160862:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828543.160866:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828543.160871:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972981504:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828543.671460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828543.671460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828543.927496:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828544.695456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828544.695457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828544.951618:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828545.719498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828545.719499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828545.975559:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828546.743469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828546.743469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828546.999571:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828547.767453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828547.767453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828548.023674:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828548.279527:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828548.279540:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828548.279545:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828548.279558:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 4998 00000100:00080000:3.0:1787828548.279573:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828548.279582:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828548.279633:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972981632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828548.279682:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828548.279689:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828548.279692:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828548.279700:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4998 00000100:00080000:3.0:1787828548.279707:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828548.279715:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828548.279733:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828548.279761:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972981760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828548.279769:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972981632:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828548.279800:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828548.279807:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828548.279811:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828548.279821:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 4998 00000100:00080000:3.0:1787828548.279829:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828548.279838:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828548.279849:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828548.279866:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828548.279881:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972981888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828548.279906:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828548.279911:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828548.279914:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828548.279920:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4998 00000100:00100000:1.0:1787828548.279926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff9c1a7895fc00 x1874668972981632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828548.279931:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828548.279973:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828548.279987:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828548.280005:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972981632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828548.280023:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828548.280032:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972982016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828548.280034:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828548.280047:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972981632:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828548.280079:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828548.280087:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972981760:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828548.280088:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828548.280096:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828548.280106:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4998 00000100:00080000:3.0:1787828548.280112:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828548.280119:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828548.280127:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828548.280136:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828548.280149:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972982144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828548.280161:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff9c1a7895e940 x1874668972981760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828548.280174:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828548.280186:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828548.280193:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828548.280218:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4998 00000100:00100000:1.0:1787828548.280225:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972981760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828548.280227:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828548.280242:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828548.280257:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828548.280258:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828548.280272:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828548.280283:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972982272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828548.280284:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972981760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828548.280305:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972981888:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828548.280318:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828548.280324:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828548.280328:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828548.280336:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 4998 00000100:00080000:3.0:1787828548.280343:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828548.280389:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828548.280409:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828548.280412:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff9c1a7895c3c0 x1874668972981888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828548.280429:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828548.280437:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972981888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828548.280451:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828548.280461:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828548.280466:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972982400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828548.280468:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972981888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828548.280488:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972982016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828548.280543:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828548.280569:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff9c1a7895cf00 x1874668972982016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828548.280594:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972982016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828548.280624:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828548.280628:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828548.280632:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972982016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828548.280656:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972982144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828548.280686:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828548.280700:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff9c1a7895f480 x1874668972982144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828548.280714:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972982144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828548.280725:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828548.280728:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828548.280732:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972982144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828548.280739:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972982272:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828548.280759:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828548.280770:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff9c1a7895d2c0 x1874668972982272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828548.280784:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972982272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828548.280793:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828548.280796:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828548.280801:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972982272:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828548.280808:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972982400:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828548.280826:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828548.280836:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828548/real 1787828548] req@ffff9c1a7895c000 x1874668972982400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828548.280849:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972982400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828548.280859:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828548.280862:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828548.280866:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972982400:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828548.791488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828548.791488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828549.047584:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828549.815463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828549.815478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828550.071531:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828550.839453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828550.839461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828551.095469:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828551.863516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828551.863520:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828552.119548:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828552.887488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828552.887506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828553.143609:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828553.399472:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828553.399480:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828553.399484:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828553.399497:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5003 00000100:00080000:0.0:1787828553.399507:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828553.399513:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828553.399543:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972982528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828553.399565:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828553.399568:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828553.399569:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828553.399572:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5003 00000100:00080000:0.0:1787828553.399575:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828553.399577:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828553.399584:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828553.399599:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972982656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828553.399607:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828553.399609:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828553.399611:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828553.399613:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5003 00000100:00080000:0.0:1787828553.399617:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828553.399622:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828553.399623:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972982528:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787828553.399625:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828553.399637:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972982784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828553.399644:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828553.399647:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828553.399648:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828553.399651:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5003 00000100:00080000:0.0:1787828553.399653:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828553.399655:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828553.399658:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828553.399667:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972982912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828553.399673:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828553.399675:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828553.399677:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828553.399680:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5003 00000100:00080000:0.0:1787828553.399682:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828553.399684:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787828553.399689:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787828553.399690:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828553.399707:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972983040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828553.399723:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828553.399726:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff9c1a78860000 x1874668972982528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828553.399730:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828553.399736:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828553.399743:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5003 00000100:00100000:1.0:1787828553.399747:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972982528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828553.399748:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828553.399758:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828553.399760:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828553.399765:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828553.399766:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828553.399773:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972982528:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828553.399779:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972982656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828553.399784:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972983168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828553.399792:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828553.399794:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828553.399796:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828553.399801:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5003 00000800:00000100:1.0:1787828553.399802:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828553.399807:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828553.399812:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828553.399816:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff9c1a78863c00 x1874668972982656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828553.399817:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828553.399830:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972983296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828553.399832:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972982656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.399854:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.399857:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828553.399860:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972982656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828553.399879:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972982784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828553.399891:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828553.399897:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff9c1a788630c0 x1874668972982784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828553.399905:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972982784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.399924:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.399928:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828553.399931:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972982784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828553.399935:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972982912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828553.399950:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828553.399959:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff9c1a78861680 x1874668972982912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828553.399966:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972982912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.399972:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.399974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828553.399976:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972982912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828553.399979:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972983040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828553.399989:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828553.399995:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff9c1a78862580 x1874668972983040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828553.400002:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972983040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.400007:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.400009:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828553.400011:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972983040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828553.400014:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972983168:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828553.400024:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828553.400030:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff9c1a78861e00 x1874668972983168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828553.400042:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972983168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.400047:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.400048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828553.400050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972983168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828553.400054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972983296:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828553.400063:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828553.400068:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828553/real 1787828553] req@ffff9c1a78860780 x1874668972983296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828553.400075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972983296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828553.400080:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828553.400082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828553.400083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972983296:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828553.911473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828553.911478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828554.167590:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828554.935457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828554.935470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828555.191644:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828555.959481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828555.959520:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828556.215559:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828556.983472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828556.983537:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828557.239603:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828558.007421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828558.007421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828558.263505:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828558.519512:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828558.519522:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828558.519527:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828558.519538:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5008 00000100:00080000:3.0:1787828558.519550:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828558.519558:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828558.519598:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972983424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828558.519633:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828558.519639:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828558.519642:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828558.519649:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5008 00000100:00080000:3.0:1787828558.519654:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828558.519660:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828558.519699:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828558.519719:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972983552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828558.519736:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828558.519742:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828558.519745:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828558.519752:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5008 00000100:00080000:3.0:1787828558.519758:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828558.519783:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828558.519784:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972983424:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787828558.519791:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828558.519808:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972983680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828558.519822:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828558.519825:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828558.519827:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828558.519831:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5008 00000100:00080000:3.0:1787828558.519836:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828558.519840:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828558.519845:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828558.519857:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972983808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828558.519868:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828558.519899:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828558.519922:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828558.519926:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828558.519932:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5008 00000100:00080000:3.0:1787828558.519937:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828558.519941:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828558.519948:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828558.519960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828558/real 1787828558] req@ffff9c1a7895e580 x1874668972983424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828558.519965:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972983936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828558.520000:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828558.520003:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828558.520005:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828558.520009:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972983424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828558.520014:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5008 00000100:00080000:3.0:1787828558.520022:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828558.520031:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828558.520037:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828558.520038:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828558.520049:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828558.520055:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972984064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828558.520057:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972983424:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828558.520077:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972983552:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828558.520079:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828558.520088:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828558.520090:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828558.520094:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5008 00000100:00080000:3.0:1787828558.520097:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828558.520101:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828558.520107:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828558.520115:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828558.520121:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972984192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828558.520133:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828558/real 1787828558] req@ffff9c1a7895c000 x1874668972983552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828558.520149:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972983552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828558.520159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828558.520163:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828558.520166:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972983552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828558.520190:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972983680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828558.520207:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828558.520216:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828558/real 1787828558] req@ffff9c1a7895d2c0 x1874668972983680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828558.520226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972983680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828558.520240:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828558.520243:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828558.520246:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972983680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828558.520251:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972983808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828558.520264:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828558.520273:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828558/real 1787828558] req@ffff9c1a7895f480 x1874668972983808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828558.520282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972983808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828558.520290:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828558.520292:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828558.520295:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972983808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828558.520299:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972983936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828558.520323:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828558.520331:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828558/real 1787828558] req@ffff9c1a7895cf00 x1874668972983936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828558.520341:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972983936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828558.520348:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828558.520350:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828558.520353:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972983936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828558.520358:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972984064:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828558.520438:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828558.520451:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828558/real 1787828558] req@ffff9c1a7895c3c0 x1874668972984064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828558.520461:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972984064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828558.520469:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828558.520472:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828558.520475:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972984064:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828558.520481:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972984192:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828558.520497:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828558.520506:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828558/real 1787828558] req@ffff9c1a7895e940 x1874668972984192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828558.520517:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972984192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828558.520525:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828558.520527:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828558.520530:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972984192:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828559.031492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828559.031492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828559.287530:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828560.055460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828560.055541:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828560.311591:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828561.079526:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828561.079526:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828561.335632:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828562.103446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828562.103446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828562.359503:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828563.127507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828563.127508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828563.383479:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828563.639434:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828563.639445:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828563.639451:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828563.639463:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5013 00000100:00080000:3.0:1787828563.639476:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828563.639485:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828563.639542:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972984320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828563.639577:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828563.639581:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828563.639582:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828563.639586:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5013 00000100:00080000:3.0:1787828563.639589:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828563.639593:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828563.639605:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828563.639617:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972984448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828563.639629:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828563.639632:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828563.639634:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828563.639640:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5013 00000100:00080000:3.0:1787828563.639644:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828563.639649:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828563.639658:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828563.639673:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972984320:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828563.639682:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972984576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828563.639706:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828563.639711:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828563.639713:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828563.639717:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5013 00000100:00080000:3.0:1787828563.639721:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828563.639725:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828563.639733:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828563.639747:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972984704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828563.639756:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828563.639760:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828563.639761:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828563.639765:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5013 00000100:00080000:3.0:1787828563.639770:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828563.639784:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828563.639869:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828563/real 1787828563] req@ffff9c1a7895fc00 x1874668972984320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828563.639877:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828563.639892:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828563.639912:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972984832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828563.639914:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972984320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828563.639965:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828563.639986:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828563.639992:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828563.639997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972984320:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828563.640011:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972984448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828563.640045:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828563.640065:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828563/real 1787828563] req@ffff9c1a7895de00 x1874668972984448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828563.640082:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972984448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828563.640104:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828563.640115:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828563.640117:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828563.640125:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828563.640125:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5013 00000100:00100000:2.0:1787828563.640147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972984448:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828563.640161:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828563.640210:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828563.640215:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972984576:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787828563.640237:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828563.640258:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972984960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828563.640272:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828563.640275:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828563.640277:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828563.640297:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828563.640304:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5013 00000100:00080000:3.0:1787828563.640316:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828563.640323:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828563.640330:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828563.640335:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828563/real 1787828563] req@ffff9c1a7895c780 x1874668972984576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828563.640347:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972985088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828563.640429:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972984576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828563.640441:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828563.640445:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828563.640450:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972984576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828563.640460:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972984704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828563.640487:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828563.640509:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828563/real 1787828563] req@ffff9c1a7895f840 x1874668972984704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828563.640534:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972984704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828563.640551:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828563.640557:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828563.640564:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972984704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828563.640577:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972984832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828563.640609:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828563.640624:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828563/real 1787828563] req@ffff9c1a7895da40 x1874668972984832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828563.640638:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972984832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828563.640649:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828563.640653:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828563.640656:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972984832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828563.640677:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972984960:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828563.640697:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828563.640708:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828563/real 1787828563] req@ffff9c1a7895cb40 x1874668972984960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828563.640722:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972984960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828563.640731:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828563.640734:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828563.640738:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972984960:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828563.640745:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972985088:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828563.640762:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828563.640772:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828563/real 1787828563] req@ffff9c1a7895f0c0 x1874668972985088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828563.640785:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972985088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828563.640795:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828563.640798:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828563.640801:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972985088:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828564.151441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828564.151464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828564.407510:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828565.175467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828565.175476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828565.431545:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828566.199468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828566.199550:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828566.455563:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828567.223431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828567.223457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828567.479583:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828568.247468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828568.247562:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828568.503634:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828568.759566:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828568.759598:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828568.759604:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828568.759617:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5019 00000100:00080000:3.0:1787828568.759633:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828568.759643:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828568.759699:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972985216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828568.759744:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828568.759748:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828568.759751:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828568.759755:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5019 00000100:00080000:3.0:1787828568.759759:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828568.759764:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828568.759778:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828568.759795:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972985344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828568.759831:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828568.759833:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972985216:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828568.759858:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828568.759862:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828568.759869:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5019 00000100:00080000:3.0:1787828568.759876:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828568.759883:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828568.759890:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828568.759909:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972985472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828568.759924:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828568.759930:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828568.759950:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828568.759955:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5019 00000100:00080000:3.0:1787828568.759961:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828568.759966:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828568.759973:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828568.759985:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972985600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828568.759998:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972985344:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828568.760003:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828568.760015:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828568.760036:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828568.760043:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5019 00000100:00080000:3.0:1787828568.760062:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828568.760070:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828568.760079:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828568.760082:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972985472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828568.760121:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972985728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828568.760134:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828568.760138:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828568.760140:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828568.760146:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5019 00000100:00080000:3.0:1787828568.760168:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828568.760171:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972985600:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828568.760178:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828568.760197:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828568.760223:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972985856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828568.760247:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972985728:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828568.760281:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828568.760290:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828568.760291:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972985856:192.168.203.148@tcp:8:kworker.0 00000100:00080000:3.0:1787828568.760300:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828568.760314:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5019 00000100:00080000:3.0:1787828568.760321:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828568.760329:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828568.760338:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828568.760374:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972985984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828568.760428:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972985984:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828569.271437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828569.271437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828569.527568:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828570.295482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828570.295515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828570.551592:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828571.319481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828571.319494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828571.575624:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828571.831749:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787828571.831835:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828571.831859:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828571.831913:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828571.831927:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828571.831940:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828571.831957:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828571.831972:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787828571.831986:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787828571.831997:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828568/real 1787828571] req@ffff9c1a7895da40 x1874668972985600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.832046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972985600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.832064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.832073:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828571.832079:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972985600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828571.832091:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828568/real 1787828571] req@ffff9c1a7895f840 x1874668972985728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.832111:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972985728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.832124:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.832131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828571.832136:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972985728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828571.832147:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828568/real 1787828571] req@ffff9c1a7895f0c0 x1874668972985344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.832165:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972985344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.832200:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.832206:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828571.832212:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972985344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828571.832227:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828568/real 1787828571] req@ffff9c1a7895cb40 x1874668972985472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.832244:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972985472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.832256:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.832262:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828571.832267:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972985472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828571.832277:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828568/real 1787828571] req@ffff9c1a7895e1c0 x1874668972985216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.832296:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972985216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.832309:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.832316:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828571.832321:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972985216:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828571.832408:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828568/real 1787828571] req@ffff9c1a7895de00 x1874668972985984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.832430:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972985984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.832443:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.832450:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828571.832455:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972985984:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828571.832466:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828568/real 1787828571] req@ffff9c1a7895c780 x1874668972985856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828571.832485:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972985856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828571.832497:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828571.832503:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828571.832509:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972985856:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828572.343415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828572.343439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828572.599431:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828573.367458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828573.367458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828573.623605:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828573.879527:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828573.879540:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828573.879545:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828573.879570:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5024 00000100:00080000:3.0:1787828573.879586:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828573.879599:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828573.879653:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972986112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828573.879703:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828573.879708:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828573.879710:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828573.879716:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5024 00000100:00080000:3.0:1787828573.879722:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828573.879727:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828573.879741:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828573.879758:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972986240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828573.879807:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828573.879811:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828573.879814:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828573.879820:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5024 00000100:00080000:3.0:1787828573.879826:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828573.879831:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828573.879838:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828573.879862:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972986112:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828573.879864:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972986368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828573.879894:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828573.879901:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828573.879905:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828573.879912:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5024 00000100:00080000:3.0:1787828573.879940:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828573.879962:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828573.879967:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828573.879981:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828573.880012:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972986496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828573.880026:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828573/real 1787828573] req@ffff9c1a7895fc00 x1874668972986112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828573.880036:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828573.880052:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828573.880059:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828573.880072:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5024 00000100:00100000:2.0:1787828573.880088:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972986112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828573.880090:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828573.880105:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828573.880122:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828573.880128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828573.880141:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828573.880151:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972986624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828573.880155:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972986112:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828573.880182:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972986240:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828573.880193:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828573.880203:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828573.880207:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828573.880217:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5024 00000100:00080000:3.0:1787828573.880226:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828573.880238:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828573.880248:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828573.880249:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828573.880286:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828573/real 1787828573] req@ffff9c1a7895e940 x1874668972986240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828573.880291:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972986752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828573.880395:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972986240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828573.880414:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828573.880418:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828573.880431:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828573.880433:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828573.880441:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828573.880449:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972986240:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828573.880459:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5024 00000100:00080000:3.0:1787828573.880471:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828573.880480:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828573.880490:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828573.880509:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972986368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828573.880522:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972986880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828573.880569:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828573.880599:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828573/real 1787828573] req@ffff9c1a7895c3c0 x1874668972986368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828573.880625:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972986368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828573.880653:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828573.880660:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828573.880668:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972986368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828573.880682:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972986496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828573.880721:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828573.880743:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828573/real 1787828573] req@ffff9c1a7895cf00 x1874668972986496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828573.880763:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972986496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828573.880809:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828573.880815:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828573.880822:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972986496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828573.880834:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972986624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828573.880886:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828573.880908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828573/real 1787828573] req@ffff9c1a7895f480 x1874668972986624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828573.880933:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972986624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828573.880950:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828573.880954:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828573.880959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972986624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828573.880967:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972986752:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828573.880998:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828573.881011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828573/real 1787828573] req@ffff9c1a7895d2c0 x1874668972986752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828573.881025:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972986752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828573.881035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828573.881038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828573.881042:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972986752:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828573.881073:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972986880:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828573.881109:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828573.881131:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828573/real 1787828573] req@ffff9c1a7895c000 x1874668972986880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828573.881156:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972986880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828573.881172:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828573.881179:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828573.881185:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972986880:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828574.391408:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828574.391413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828574.647541:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828575.415443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828575.415443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828575.671532:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828576.439453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828576.439465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828576.695521:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828577.463440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828577.463444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828577.719504:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828578.487472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828578.487492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828578.743565:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828578.999440:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828578.999452:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828578.999459:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828578.999475:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5029 00000100:00080000:3.0:1787828578.999507:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828578.999520:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828578.999577:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972987008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828578.999617:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828578.999622:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828578.999625:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828578.999631:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5029 00000100:00080000:3.0:1787828578.999637:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828578.999643:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828578.999656:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828578.999676:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972987136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828578.999693:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972987008:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828578.999697:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828578.999710:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828578.999713:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828578.999759:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5029 00000100:00080000:3.0:1787828578.999767:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828578.999774:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828578.999782:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828578.999801:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972987264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828578.999824:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828578.999828:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828578.999839:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828578.999843:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828578.999854:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5029 00000100:00080000:3.0:1787828578.999862:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828578.999872:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828578.999884:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828578.999886:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828578/real 1787828578] req@ffff9c1a7895e580 x1874668972987008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828578.999918:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972987392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828578.999922:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972987008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828578.999978:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828578.999982:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828578.999986:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828579.000000:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972987008:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828579.000000:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828579.000011:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828579.000026:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5029 00000100:00080000:3.0:1787828579.000036:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828579.000044:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972987136:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828579.000049:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828579.000062:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828579.000089:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972987520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828579.000093:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828579.000117:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828578/real 1787828579] req@ffff9c1a7895c000 x1874668972987136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828579.000123:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828579.000150:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972987136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828579.000159:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828579.000168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828579.000170:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828579.000180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828579.000187:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5029 00000100:00100000:2.0:1787828579.000192:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972987136:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828579.000200:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828579.000206:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972987264:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828579.000218:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828579.000236:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828579.000247:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828579.000261:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828578/real 1787828579] req@ffff9c1a7895d2c0 x1874668972987264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.000282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972987264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828579.000290:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972987648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828579.000327:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828579.000329:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828579.000374:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828579.000378:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828579.000384:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5029 00000100:00080000:3.0:1787828579.000391:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828579.000397:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828579.000405:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828579.000413:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828579.000424:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972987776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828579.000427:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972987264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828579.000454:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972987392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828579.000510:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828579.000527:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828579/real 1787828579] req@ffff9c1a7895f480 x1874668972987392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.000542:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972987392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.000553:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.000557:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828579.000561:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972987392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828579.000583:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972987520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828579.000605:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828579.000617:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828579/real 1787828579] req@ffff9c1a7895cf00 x1874668972987520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.000641:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972987520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.000651:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.000655:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828579.000658:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972987520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828579.000665:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972987648:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828579.000685:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828579.000696:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828579/real 1787828579] req@ffff9c1a7895c3c0 x1874668972987648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.000710:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972987648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.000720:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.000723:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828579.000727:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972987648:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828579.000763:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972987776:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828579.000794:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828579.000809:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828579/real 1787828579] req@ffff9c1a7895e940 x1874668972987776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828579.000824:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972987776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828579.000843:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828579.000846:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828579.000850:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972987776:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828579.511514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828579.511515:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828579.767510:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828580.535433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828580.535433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828580.791488:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828581.559409:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828581.559421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828581.815548:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828582.583433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828582.583433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828582.839484:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828583.607450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828583.607511:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828583.863505:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828584.119431:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828584.119440:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828584.119443:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828584.119451:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5034 00000100:00080000:3.0:1787828584.119460:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828584.119466:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828584.119498:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972987904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828584.119521:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828584.119523:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828584.119525:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828584.119528:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5034 00000100:00080000:3.0:1787828584.119530:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828584.119533:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828584.119540:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828584.119548:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972988032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828584.119555:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828584.119557:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828584.119558:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828584.119561:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5034 00000100:00080000:3.0:1787828584.119562:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828584.119564:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828584.119567:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828584.119575:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972988160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828584.119587:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828584.119590:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828584.119591:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828584.119593:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5034 00000100:00080000:3.0:1787828584.119595:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828584.119599:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972987904:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828584.119600:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828584.119607:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828584.119614:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972988288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828584.119621:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828584.119623:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828584.119625:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828584.119627:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5034 00000100:00080000:3.0:1787828584.119629:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828584.119632:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828584.119634:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828584.119642:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972988416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828584.119647:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828584.119649:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828584.119651:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828584.119654:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5034 00000100:00080000:3.0:1787828584.119656:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828584.119659:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828584.119664:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828584.119665:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828584.119674:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972988544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828584.119681:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828584.119684:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828584.119685:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828584.119687:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5034 00000100:00080000:3.0:1787828584.119689:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828584.119692:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828584.119695:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828584.119716:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff9c1a7895fc00 x1874668972987904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828584.119720:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972988672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828584.119747:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972987904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.119756:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.119760:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828584.119764:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972987904:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828584.119770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972988032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828584.119786:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828584.119794:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff9c1a7895c780 x1874668972988032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.119802:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972988032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.119809:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.119811:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828584.119814:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972988032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828584.119817:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972988160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828584.119828:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828584.119834:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff9c1a7895de00 x1874668972988160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.119841:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972988160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.119846:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.119848:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828584.119850:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972988160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828584.119870:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972988288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828584.119881:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828584.119888:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff9c1a7895e1c0 x1874668972988288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.119895:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972988288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.119900:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.119902:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828584.119905:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972988288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828584.119909:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972988416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828584.119919:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828584.119924:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff9c1a7895cb40 x1874668972988416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.119931:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972988416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.119936:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.119938:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828584.119939:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972988416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828584.119943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972988544:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828584.119953:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828584.119959:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff9c1a7895f0c0 x1874668972988544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.119967:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972988544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.119972:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.119974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828584.119975:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972988544:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828584.119979:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972988672:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828584.119988:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828584.119994:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828584/real 1787828584] req@ffff9c1a7895f840 x1874668972988672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828584.120002:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972988672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828584.120015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828584.120017:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828584.120019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972988672:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828584.631444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828584.631452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828584.887502:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828585.655411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828585.655411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828585.911464:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828586.679438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828586.679452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828586.935444:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828587.703434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828587.703449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828587.959486:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828588.727452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828588.727458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828588.983516:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828589.239471:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828589.239500:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828589.239504:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828589.239513:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5039 00000100:00080000:3.0:1787828589.239523:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828589.239530:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828589.239567:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972988800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828589.239623:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828589.239628:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828589.239631:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828589.239637:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5039 00000100:00080000:3.0:1787828589.239643:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828589.239648:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828589.239685:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828589.239733:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972988928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828589.239738:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972988800:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828589.239750:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828589.239771:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828589.239774:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828589.239790:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5039 00000100:00080000:3.0:1787828589.239796:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828589.239802:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828589.239809:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828589.239823:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972989056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828589.239834:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828589.239836:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828589.239838:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828589.239841:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5039 00000100:00080000:3.0:1787828589.239846:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828589.239850:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828589.239854:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828589.239862:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828589.239864:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972989184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828589.239874:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828589.239878:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828589.239880:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828589.239883:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5039 00000100:00080000:3.0:1787828589.239894:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828589.239898:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828589.239902:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828589.239937:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff9c1a7895da40 x1874668972988800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828589.239938:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972989312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828589.240021:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828589.240028:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828589.240032:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828589.240042:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5039 00000100:00100000:2.0:1787828589.240045:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972988800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828589.240068:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828589.240075:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828589.240083:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828589.240085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828589.240090:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828589.240094:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972989440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828589.240098:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972988800:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828589.240115:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828589.240135:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828589.240140:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972988928:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828589.240141:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828589.240157:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5039 00000100:00080000:3.0:1787828589.240171:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828589.240178:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828589.240208:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828589.240217:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828589.240228:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972989568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828589.240240:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff9c1a7895f840 x1874668972988928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828589.240283:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972988928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828589.240291:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828589.240295:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828589.240298:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972988928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828589.240304:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972989056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828589.240337:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828589.240406:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff9c1a7895f0c0 x1874668972989056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828589.240421:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972989056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828589.240431:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828589.240435:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828589.240439:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972989056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828589.240484:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972989184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828589.240515:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828589.240541:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff9c1a7895cb40 x1874668972989184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828589.240549:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972989184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828589.240560:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828589.240562:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828589.240564:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972989184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828589.240568:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972989312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828589.240580:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828589.240585:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff9c1a7895e1c0 x1874668972989312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828589.240593:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972989312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828589.240598:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828589.240600:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828589.240602:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972989312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828589.240606:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972989440:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828589.240617:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828589.240622:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff9c1a7895de00 x1874668972989440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828589.240630:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972989440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828589.240635:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828589.240637:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828589.240639:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972989440:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828589.240655:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972989568:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828589.240684:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828589.240693:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828589/real 1787828589] req@ffff9c1a7895c780 x1874668972989568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828589.240701:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972989568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828589.240707:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828589.240709:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828589.240711:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972989568:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828589.751429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828589.751435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828590.007531:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828590.775428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828590.775428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828591.031519:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828591.799414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828591.799416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828592.055457:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828592.823468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828592.823475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828593.079530:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828593.847469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828593.847477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828594.103451:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828594.359478:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828594.359487:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828594.359490:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828594.359498:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5044 00000100:00080000:0.0:1787828594.359507:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828594.359513:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828594.359544:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972989696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828594.359567:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828594.359570:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828594.359571:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828594.359575:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5044 00000100:00080000:0.0:1787828594.359578:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828594.359581:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828594.359596:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828594.359607:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972989696:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828594.359611:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972989824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828594.359621:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828594.359624:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828594.359626:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828594.359641:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5044 00000100:00080000:0.0:1787828594.359644:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828594.359648:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828594.359652:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828594.359662:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972989952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828594.359670:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828594.359672:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828594.359674:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828594.359679:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828594.359682:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5044 00000100:00080000:0.0:1787828594.359686:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828594.359690:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828594.359693:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828594.359702:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972990080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828594.359713:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828594.359715:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828594.359718:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff9c1a788621c0 x1874668972989696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828594.359719:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828594.359727:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5044 00000100:00080000:0.0:1787828594.359733:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828594.359738:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828594.359741:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828594.359745:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972989696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828594.359752:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972990208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828594.359757:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828594.359762:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828594.359766:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828594.359770:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828594.359773:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972989696:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828594.359775:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828594.359784:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5044 00000100:00080000:0.0:1787828594.359789:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828594.359794:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828594.359799:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972989824:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787828594.359801:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828594.359820:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972990336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828594.359824:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828594.359837:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828594.359840:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff9c1a78862940 x1874668972989824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828594.359841:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828594.359845:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828594.359853:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5044 00000100:00080000:0.0:1787828594.359859:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828594.359863:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972989824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828594.359867:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828594.359874:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828594.359877:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.359882:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828594.359887:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972990464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828594.359888:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972989824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828594.359900:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972989952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828594.359918:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828594.359929:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff9c1a78861a40 x1874668972989952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828594.359937:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972989952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828594.359943:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.359945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828594.359947:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972989952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828594.359951:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972990080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828594.359962:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828594.359968:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff9c1a78860f00 x1874668972990080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828594.359976:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972990080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828594.359981:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.359983:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828594.359984:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972990080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828594.359988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972990208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828594.359998:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828594.360003:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff9c1a78862d00 x1874668972990208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828594.360010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972990208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828594.360016:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.360017:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828594.360019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972990208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828594.360031:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972990336:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828594.360042:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828594.360048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff9c1a78860b40 x1874668972990336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828594.360060:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972990336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828594.360066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.360067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828594.360069:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972990336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828594.360073:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972990464:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828594.360082:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828594.360087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828594/real 1787828594] req@ffff9c1a78863840 x1874668972990464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828594.360094:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972990464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828594.360100:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828594.360101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828594.360103:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972990464:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828594.871471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828594.871473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828595.127524:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828595.895402:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828595.895408:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828596.151523:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828596.919434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828596.919437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828597.175541:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828597.943444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828597.943444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828598.199512:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828598.967449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828598.967449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828599.223555:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828599.479520:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828599.479529:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828599.479533:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828599.479543:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5049 00000100:00080000:3.0:1787828599.479553:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828599.479558:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828599.479588:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972990592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828599.479623:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828599.479626:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828599.479627:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828599.479632:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5049 00000100:00080000:3.0:1787828599.479635:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828599.479638:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828599.479647:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828599.479662:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972990720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828599.479673:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828599.479679:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828599.479681:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972990592:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828599.479681:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828599.479700:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5049 00000100:00080000:3.0:1787828599.479703:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828599.479706:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828599.479711:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828599.479725:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972990848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828599.479735:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828599.479738:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828599.479739:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828599.479745:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5049 00000100:00080000:3.0:1787828599.479766:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828599.479772:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828599.479776:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828599.479824:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828599/real 1787828599] req@ffff9c1a7895fc00 x1874668972990592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828599.479849:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972990592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828599.479863:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828599.479865:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828599.479875:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828599.479880:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972990592:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828599.479914:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972990720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828599.479943:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828599.479957:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828599/real 1787828599] req@ffff9c1a7895e940 x1874668972990720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828599.479967:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972990720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828599.479974:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828599.479978:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828599.479994:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972990720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828599.479996:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972990976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828599.480010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972990848:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828599.480017:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828599.480021:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828599.480023:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828599.480027:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5049 00000100:00080000:3.0:1787828599.480032:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828599.480038:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828599.480063:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828599/real 1787828599] req@ffff9c1a7895c3c0 x1874668972990848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828599.480080:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972990848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828599.480093:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828599.480099:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828599.480103:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828599.480111:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828599.480112:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972990848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828599.480129:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972991104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828599.480138:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828599.480145:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972990976:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828599.480147:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828599.480151:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828599.480161:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5049 00000100:00080000:3.0:1787828599.480165:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828599.480168:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828599.480172:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828599.480184:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972991232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828599.480191:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828599.480204:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828599.480207:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828599.480209:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828599.480211:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828599/real 1787828599] req@ffff9c1a7895cf00 x1874668972990976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828599.480216:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5049 00000100:00080000:3.0:1787828599.480224:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828599.480233:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828599.480237:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972990976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828599.480240:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828599.480254:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828599.480255:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972991360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828599.480268:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828599.480283:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972990976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828599.480302:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972991104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828599.480328:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828599.480341:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828599/real 1787828599] req@ffff9c1a7895f480 x1874668972991104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828599.480388:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972991104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828599.480396:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828599.480399:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828599.480404:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972991104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828599.480411:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972991232:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828599.480435:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828599.480447:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828599/real 1787828599] req@ffff9c1a7895d2c0 x1874668972991232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828599.480455:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972991232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828599.480461:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828599.480463:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828599.480465:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972991232:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828599.480469:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972991360:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828599.480479:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828599.480485:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828599/real 1787828599] req@ffff9c1a7895c000 x1874668972991360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828599.480492:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972991360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828599.480497:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828599.480498:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828599.480500:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972991360:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828599.991507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828599.991513:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828600.247502:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828601.015424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828601.015430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828601.271458:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828602.039488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828602.039488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828602.295498:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828603.063483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828603.063483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828603.319489:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828604.087506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828604.087507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828604.343551:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828604.599424:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828604.599436:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828604.599441:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828604.599450:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5054 00000100:00080000:3.0:1787828604.599462:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828604.599469:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828604.599513:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972991488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828604.599538:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828604.599541:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828604.599542:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828604.599545:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5054 00000100:00080000:3.0:1787828604.599548:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828604.599550:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828604.599571:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828604.599584:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972991616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828604.599598:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828604.599601:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828604.599602:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828604.599606:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5054 00000100:00080000:3.0:1787828604.599609:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828604.599612:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828604.599616:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828604.599619:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972991488:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828604.599626:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972991744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828604.599635:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828604.599638:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828604.599639:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828604.599642:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5054 00000100:00080000:3.0:1787828604.599644:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828604.599646:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828604.599649:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828604.599657:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972991872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828604.599664:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828604.599666:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828604.599667:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828604.599670:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5054 00000100:00080000:3.0:1787828604.599683:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828604.599686:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828604.599690:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828604.599695:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828604.599700:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972992000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828604.599708:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828604.599711:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828604.599713:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828604.599715:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5054 00000100:00080000:3.0:1787828604.599718:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828604.599720:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828604.599724:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828604.599731:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972992128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828604.599738:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828604.599741:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828604/real 1787828604] req@ffff9c1a7895e580 x1874668972991488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828604.599743:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828604.599748:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828604.599764:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5054 00000100:00080000:3.0:1787828604.599770:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828604.599772:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972991488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828604.599779:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828604.599786:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828604.599787:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828604.599792:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828604.599796:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972991488:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828604.599807:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972991616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828604.599808:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972992256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828604.599830:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828604.599843:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828604/real 1787828604] req@ffff9c1a7895c000 x1874668972991616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828604.599851:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972991616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828604.599857:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828604.599859:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828604.599861:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972991616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828604.599887:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972991744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828604.599906:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828604.599918:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828604/real 1787828604] req@ffff9c1a7895d2c0 x1874668972991744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828604.599931:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972991744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828604.599939:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828604.599941:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828604.599943:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972991744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828604.599948:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972991872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828604.599963:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828604.599971:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828604/real 1787828604] req@ffff9c1a7895f480 x1874668972991872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828604.599978:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972991872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828604.599983:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828604.599985:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828604.599987:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972991872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828604.599991:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972992000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828604.600001:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828604.600006:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828604/real 1787828604] req@ffff9c1a7895cf00 x1874668972992000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828604.600018:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972992000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828604.600023:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828604.600025:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828604.600027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972992000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828604.600030:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972992128:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828604.600041:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828604.600046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828604/real 1787828604] req@ffff9c1a7895c3c0 x1874668972992128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828604.600054:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972992128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828604.600058:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828604.600060:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828604.600062:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972992128:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828604.600065:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972992256:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828604.600080:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828604.600090:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828604/real 1787828604] req@ffff9c1a7895e940 x1874668972992256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828604.600102:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972992256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828604.600117:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828604.600121:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828604.600124:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972992256:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828605.111440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828605.111441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828605.367538:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828606.135448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828606.135464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828606.391576:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828607.159446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828607.159466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828607.415493:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828608.183516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828608.183516:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828608.439572:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828609.207439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828609.207448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828609.463571:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828609.719496:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828609.719507:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828609.719511:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828609.719535:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5060 00000100:00080000:0.0:1787828609.719547:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828609.719555:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828609.719593:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668972992384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828609.719630:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828609.719635:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828609.719637:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828609.719642:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5060 00000100:00080000:0.0:1787828609.719646:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828609.719650:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828609.719659:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828609.719675:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668972992512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828609.719689:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972992384:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828609.719700:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828609.719705:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828609.719706:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828609.719712:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5060 00000100:00080000:0.0:1787828609.719717:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828609.719721:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828609.719726:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828609.719739:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668972992640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828609.719750:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828609.719752:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828609.719757:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828609.719760:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828609.719765:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5060 00000100:00080000:0.0:1787828609.719769:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828609.719774:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828609.719779:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828609.719791:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668972992768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828609.719798:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828609/real 1787828609] req@ffff9c1a788612c0 x1874668972992384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828609.719805:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828609.719813:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828609.719815:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828609.719820:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668972992384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828609.719822:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5060 00000100:00080000:0.0:1787828609.719831:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828609.719834:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828609.719839:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828609.719841:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828609.719847:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972992384:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787828609.719848:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828609.719864:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668972992896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828609.719873:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972992512:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828609.719876:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828609.719881:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828609.719884:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828609.719889:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5060 00000100:00080000:0.0:1787828609.719894:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828609.719894:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828609.719903:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828609.719908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828609/real 1787828609] req@ffff9c1a78863840 x1874668972992512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828609.719910:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828609.719929:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668972993024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828609.719930:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668972992512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828609.719941:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828609.719946:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828609.719950:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828609.719958:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5060 00000100:00080000:2.0:1787828609.719959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828609.719966:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828609.719967:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828609.719970:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972992512:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828609.719975:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828609.719979:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972992640:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787828609.719984:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828609.720001:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668972993152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828609.720001:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828609.720011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828609/real 1787828609] req@ffff9c1a78860b40 x1874668972992640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828609.720051:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668972992640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828609.720057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828609.720060:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828609.720063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972992640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828609.720067:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972992768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828609.720079:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828609.720085:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828609/real 1787828609] req@ffff9c1a78862d00 x1874668972992768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828609.720093:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668972992768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828609.720103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828609.720105:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828609.720107:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972992768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828609.720117:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972992896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828609.720133:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828609.720144:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828609/real 1787828609] req@ffff9c1a78860f00 x1874668972992896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828609.720156:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668972992896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828609.720164:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828609.720168:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828609.720170:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972992896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828609.720176:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972993024:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828609.720196:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828609.720206:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828609/real 1787828609] req@ffff9c1a78861a40 x1874668972993024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828609.720218:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668972993024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828609.720226:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828609.720228:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828609.720232:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972993024:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828609.720238:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972993152:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828609.720256:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828609.720264:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828609/real 1787828609] req@ffff9c1a78862940 x1874668972993152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828609.720275:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668972993152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828609.720283:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828609.720286:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828609.720289:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972993152:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828610.231417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828610.231417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828610.487559:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828611.255432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828611.255459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828611.511569:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828612.279459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828612.279467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828612.535552:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828613.303413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828613.303413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828613.559468:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828614.327425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828614.327428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828614.583528:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828614.839470:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828614.839481:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828614.839499:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828614.839510:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5065 00000100:00080000:3.0:1787828614.839522:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828614.839531:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828614.839569:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972993280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828614.839598:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828614.839602:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828614.839604:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828614.839610:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5065 00000100:00080000:3.0:1787828614.839615:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828614.839620:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828614.839629:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828614.839630:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972993280:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828614.839647:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972993408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828614.839661:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828614.839665:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828614.839667:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828614.839671:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5065 00000100:00080000:3.0:1787828614.839687:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828614.839691:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828614.839697:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828614.839698:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828614.839716:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972993536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828614.839728:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828614.839732:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828614.839734:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828614.839739:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828614/real 1787828614] req@ffff9c1a7895fc00 x1874668972993280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828614.839741:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5065 00000100:00080000:3.0:1787828614.839748:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828614.839757:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828614.839765:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828614.839769:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972993280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828614.839782:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972993664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828614.839785:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828614.839794:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828614.839800:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828614.839802:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972993280:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828614.839808:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828614.839812:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828614.839817:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5065 00000100:00080000:3.0:1787828614.839822:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828614.839826:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828614.839831:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828614.839833:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972993408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828614.839846:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972993792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828614.839862:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828614.839870:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828614.839875:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828614.839879:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828614/real 1787828614] req@ffff9c1a7895c780 x1874668972993408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828614.839879:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828614.839889:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5065 00000100:00080000:3.0:1787828614.839898:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828614.839904:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972993408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828614.839906:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828614.839916:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828614.839919:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828614.839925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828614.839930:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972993408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828614.839933:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972993920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828614.839941:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972993536:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828614.839951:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828614.839956:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828614.839959:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828614.839964:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5065 00000800:00000100:2.0:1787828614.839969:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828614.839972:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828614.839979:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828614.839987:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828614.839994:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828614/real 1787828614] req@ffff9c1a7895de00 x1874668972993536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828614.840000:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972994048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828614.840020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972993536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828614.840031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828614.840035:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828614.840039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972993536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828614.840045:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972993664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828614.840064:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828614.840075:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828614/real 1787828614] req@ffff9c1a7895e1c0 x1874668972993664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828614.840087:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972993664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828614.840096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828614.840099:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828614.840103:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972993664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828614.840118:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972993792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828614.840135:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828614.840144:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828614/real 1787828614] req@ffff9c1a7895cb40 x1874668972993792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828614.840156:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972993792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828614.840165:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828614.840169:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828614.840172:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972993792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828614.840178:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972993920:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828614.840193:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828614.840202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828614/real 1787828614] req@ffff9c1a7895f0c0 x1874668972993920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828614.840213:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972993920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828614.840223:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828614.840226:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828614.840229:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972993920:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828614.840235:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972994048:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828614.840253:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828614.840263:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828614/real 1787828614] req@ffff9c1a7895f840 x1874668972994048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828614.840275:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972994048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828614.840284:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828614.840287:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828614.840291:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972994048:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828615.351397:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828615.351449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828615.607456:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828616.375459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828616.375508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828616.631512:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828617.399400:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828617.399425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828617.655462:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828618.423420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828618.423432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828618.679536:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828619.447473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828619.447473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828619.703501:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828619.959438:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828619.959447:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828619.959465:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828619.959476:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5070 00000100:00080000:3.0:1787828619.959487:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828619.959494:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828619.959539:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972994176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828619.959566:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828619.959569:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828619.959571:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828619.959580:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5070 00000100:00080000:3.0:1787828619.959583:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828619.959591:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828619.959601:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828619.959617:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972994304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828619.959623:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972994176:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828619.959634:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828619.959638:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828619.959639:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828619.959643:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5070 00000100:00080000:3.0:1787828619.959647:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828619.959649:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828619.959653:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828619.959662:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972994432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828619.959682:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828619.959685:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828619.959686:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828619.959690:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5070 00000100:00080000:3.0:1787828619.959694:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828619.959697:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828619.959701:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828619.959708:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828619.959712:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972994560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828619.959720:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828619.959723:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828619.959725:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828619.959730:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5070 00000100:00080000:3.0:1787828619.959734:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828619.959737:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828619.959741:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828619.959750:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972994688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828619.959758:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828619.959765:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828619/real 1787828619] req@ffff9c1a7895da40 x1874668972994176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828619.959766:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828619.959772:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828619.959787:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5070 00000100:00080000:3.0:1787828619.959799:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828619.959808:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828619.959817:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828619.959825:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972994176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828619.959830:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972994816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828619.959850:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828619.959853:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828619.959860:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828619.959860:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828619.959869:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828619.959874:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972994176:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828619.959880:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5070 00000100:00080000:3.0:1787828619.959895:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828619.959901:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828619.959907:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828619.959919:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972994304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828619.959925:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972994944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828619.959945:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828619.959958:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828619/real 1787828619] req@ffff9c1a7895f840 x1874668972994304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828619.959970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972994304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828619.959980:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828619.959984:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828619.959988:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972994304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828619.959994:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972994432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828619.960015:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828619.960027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828619/real 1787828619] req@ffff9c1a7895f0c0 x1874668972994432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828619.960037:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972994432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828619.960048:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828619.960050:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828619.960052:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972994432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828619.960057:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972994560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828619.960073:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828619.960080:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828619/real 1787828619] req@ffff9c1a7895cb40 x1874668972994560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828619.960087:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972994560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828619.960093:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828619.960095:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828619.960097:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972994560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828619.960100:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972994688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828619.960111:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828619.960116:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828619/real 1787828619] req@ffff9c1a7895e1c0 x1874668972994688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828619.960124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972994688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828619.960129:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828619.960131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828619.960133:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972994688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828619.960137:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972994816:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828619.960148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828619.960153:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828619/real 1787828619] req@ffff9c1a7895de00 x1874668972994816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828619.960160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972994816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828619.960166:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828619.960168:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828619.960170:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972994816:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828619.960183:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972994944:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828619.960193:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828619.960199:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828619/real 1787828619] req@ffff9c1a7895c780 x1874668972994944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828619.960207:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972994944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828619.960213:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828619.960214:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828619.960216:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972994944:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828620.471491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828620.471509:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828620.727542:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828621.495449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828621.495451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828621.751489:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828622.519401:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828622.519407:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828622.775480:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828623.543414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828623.543415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828623.799503:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828624.567405:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828624.567410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828624.823479:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828625.079451:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828625.079462:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828625.079467:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828625.079478:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5075 00000100:00080000:3.0:1787828625.079491:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828625.079500:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828625.079548:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972995072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828625.079570:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828625.079573:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828625.079575:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828625.079579:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5075 00000100:00080000:3.0:1787828625.079584:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828625.079587:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828625.079598:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828625.079611:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972995200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828625.079612:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972995072:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828625.079641:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828625.079645:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828625.079647:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828625.079652:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5075 00000100:00080000:3.0:1787828625.079655:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828625.079658:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828625.079661:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828625.079673:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972995328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828625.079681:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828625.079684:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828625.079686:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828625.079689:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5075 00000100:00080000:3.0:1787828625.079692:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828625.079694:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828625.079698:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828625.079706:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828625.079713:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972995456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828625.079721:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828625.079724:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828625.079727:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828625.079730:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5075 00000100:00080000:3.0:1787828625.079735:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828625.079737:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828625.079741:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828625.079751:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972995584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828625.079759:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828625.079762:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff9c1a7895fc00 x1874668972995072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828625.079763:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828625.079767:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828625.079778:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5075 00000100:00080000:3.0:1787828625.079787:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828625.079798:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828625.079806:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828625.079811:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972995072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828625.079819:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972995712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828625.079829:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828625.079831:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828625.079841:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828625.079842:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828625.079852:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828625.079858:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972995072:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828625.079864:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5075 00000100:00080000:3.0:1787828625.079872:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828625.079878:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828625.079882:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828625.079895:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972995840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828625.079898:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972995200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828625.079939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828625.079953:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff9c1a7895e940 x1874668972995200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828625.079962:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972995200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828625.079968:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828625.079971:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828625.079973:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972995200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828625.079977:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972995328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828625.079990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828625.079996:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff9c1a7895c3c0 x1874668972995328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828625.080004:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972995328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828625.080009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828625.080011:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828625.080013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972995328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828625.080017:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972995456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828625.080027:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828625.080033:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff9c1a7895cf00 x1874668972995456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828625.080041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972995456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828625.080046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828625.080048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828625.080050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972995456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828625.080054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972995584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828625.080064:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828625.080070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff9c1a7895f480 x1874668972995584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828625.080082:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972995584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828625.080088:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828625.080089:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828625.080091:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972995584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828625.080095:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972995712:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828625.080122:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828625.080129:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff9c1a7895d2c0 x1874668972995712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828625.080137:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972995712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828625.080142:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828625.080145:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828625.080147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972995712:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828625.080160:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972995840:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828625.080171:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828625.080177:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828625/real 1787828625] req@ffff9c1a7895c000 x1874668972995840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828625.080185:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972995840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828625.080195:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828625.080197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828625.080199:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972995840:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828625.591501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828625.591501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828625.847574:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828626.615398:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828626.615432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828626.871550:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828627.639446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828627.639451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828627.895532:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828628.663440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828628.663464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828628.919491:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828629.687417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828629.687437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828629.943479:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828630.199458:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828630.199466:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828630.199470:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828630.199479:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5080 00000100:00080000:3.0:1787828630.199489:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828630.199495:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828630.199527:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668972995968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828630.199550:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828630.199553:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828630.199554:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828630.199557:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5080 00000100:00080000:3.0:1787828630.199560:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828630.199563:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828630.199570:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828630.199580:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668972996096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828630.199588:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828630.199590:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828630.199591:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828630.199595:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972995968:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828630.199596:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5080 00000100:00080000:3.0:1787828630.199600:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828630.199606:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828630.199609:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828630.199616:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668972996224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828630.199623:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828630.199626:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828630.199627:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828630.199630:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5080 00000100:00080000:3.0:1787828630.199632:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828630.199634:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828630.199637:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828630.199644:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668972996352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828630.199650:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828630.199652:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828630.199653:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828630.199655:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5080 00000100:00080000:3.0:1787828630.199658:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828630.199661:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828630.199664:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828630.199665:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828630.199674:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668972996480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828630.199680:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828630.199682:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828630.199684:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828630.199686:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5080 00000100:00080000:3.0:1787828630.199688:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828630.199691:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828630.199693:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828630.199700:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668972996608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828630.199706:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828630.199708:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828630.199709:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828630.199712:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff9c1a7895e580 x1874668972995968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828630.199714:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5080 00000100:00080000:3.0:1787828630.199719:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828630.199724:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828630.199730:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828630.199739:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668972996736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828630.199743:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668972995968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828630.199756:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828630.199759:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828630.199763:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972995968:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828630.199783:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972996096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828630.199802:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828630.199811:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff9c1a7895c000 x1874668972996096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828630.199824:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668972996096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828630.199830:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828630.199832:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828630.199834:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972996096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828630.199838:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972996224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828630.199848:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828630.199854:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff9c1a7895d2c0 x1874668972996224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828630.199861:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668972996224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828630.199867:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828630.199869:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828630.199870:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972996224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828630.199874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972996352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828630.199883:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828630.199889:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff9c1a7895f480 x1874668972996352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828630.199896:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668972996352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828630.199906:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828630.199908:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828630.199910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972996352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828630.199914:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972996480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828630.199923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828630.199928:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff9c1a7895cf00 x1874668972996480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828630.199935:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668972996480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828630.199940:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828630.199942:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828630.199944:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972996480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828630.199947:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972996608:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828630.199957:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828630.199963:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff9c1a7895c3c0 x1874668972996608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828630.199970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668972996608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828630.199976:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828630.199977:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828630.199980:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972996608:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828630.199983:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972996736:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828630.199992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828630.199997:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828630/real 1787828630] req@ffff9c1a7895e940 x1874668972996736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828630.200004:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668972996736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828630.200009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828630.200011:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828630.200013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972996736:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828630.711420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828630.711445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828630.967523:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828631.735459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828631.735460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828631.991583:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828632.759449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828632.759451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828633.015593:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828633.783450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828633.783450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828634.039518:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828634.807439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828634.807439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828635.063514:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828635.319466:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828635.319476:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828635.319480:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828635.319491:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5085 00000100:00080000:3.0:1787828635.319502:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828635.319509:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828635.319561:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972996864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828635.319586:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828635.319589:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828635.319590:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828635.319594:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5085 00000100:00080000:3.0:1787828635.319597:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828635.319600:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828635.319608:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828635.319618:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668972996992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828635.319629:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828635.319630:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972996864:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828635.319635:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828635.319639:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828635.319644:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5085 00000100:00080000:3.0:1787828635.319646:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828635.319648:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828635.319652:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828635.319663:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972997120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828635.319671:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828635.319674:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828635.319675:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828635.319678:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5085 00000100:00080000:3.0:1787828635.319691:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828635.319696:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828635.319700:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828635.319714:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972997248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828635.319727:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828635.319731:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828635.319734:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828635.319740:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5085 00000100:00080000:3.0:1787828635.319746:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828635.319749:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972996992:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828635.319756:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828635.319762:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828635.319777:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972997376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828635.319791:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972997120:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828635.319796:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828635.319817:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828635.319819:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828635.319826:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5085 00000100:00080000:3.0:1787828635.319832:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828635.319838:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972997248:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828635.319838:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828635.319852:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828635.319875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972997376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828635.319878:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972997504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828635.319897:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828635.319903:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828635.319906:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828635.319913:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5085 00000100:00100000:2.0:1787828635.319917:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972997504:192.168.203.148@tcp:8:kworker.0 00000100:00080000:3.0:1787828635.319921:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828635.319929:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828635.319936:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828635.319951:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972997632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828635.320026:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972997632:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828635.831460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828635.831463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828636.087526:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828636.855466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828636.855466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828637.111554:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828637.879431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828637.879431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828638.135650:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828638.391705:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787828638.391749:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828638.391779:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828638.391833:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828638.391852:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828638.391866:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787828638.391878:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828638] req@ffff9c1a7895c780 x1874668972996992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828638.391879:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828638.391899:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787828638.391920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668972996992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828638.391920:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00080000:1.0:1787828638.391940:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828638.391947:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828638.391952:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972996992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828638.391962:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828638] req@ffff9c1a7895de00 x1874668972997120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828638.391977:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972997120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828638.391986:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828638.392018:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828638.392022:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972997120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828638.392036:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828638] req@ffff9c1a7895e1c0 x1874668972997248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828638.392052:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972997248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828638.392062:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828638.392066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828638.392070:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972997248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828638.392079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828638] req@ffff9c1a7895fc00 x1874668972996864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828638.392092:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972996864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828638.392100:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828638.392104:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828638.392108:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972996864:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828638.392139:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828638] req@ffff9c1a7895f840 x1874668972997632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828638.392153:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972997632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828638.392162:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828638.392166:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828638.392170:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972997632:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828638.392177:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828638] req@ffff9c1a7895f0c0 x1874668972997504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828638.392191:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972997504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828638.392199:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828638.392203:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828638.392207:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972997504:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828638.392213:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828635/real 1787828638] req@ffff9c1a7895cb40 x1874668972997376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828638.392226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972997376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828638.392234:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828638.392238:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828638.392241:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972997376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828638.903456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828638.903487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828639.159540:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828639.927427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828639.927462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828640.183561:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828640.439452:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.439463:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828640.439467:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828640.439477:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5090 00000100:00080000:3.0:1787828640.439489:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.439496:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828640.439532:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668972997760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828640.439562:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.439566:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828640.439568:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828640.439573:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5090 00000100:00080000:3.0:1787828640.439578:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.439583:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828640.439593:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828640.439594:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972997760:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828640.439613:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668972997888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828640.439626:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.439630:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828640.439632:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828640.439636:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5090 00000100:00080000:3.0:1787828640.439642:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.439646:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828640.439651:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828640.439664:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668972998016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828640.439668:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828640.439680:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.439685:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828640.439687:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828640.439691:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5090 00000100:00080000:3.0:1787828640.439696:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.439701:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828640.439706:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828640.439722:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828640] req@ffff9c1a7895da40 x1874668972997760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828640.439728:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668972998144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828640.439752:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.439755:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828640.439760:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828640.439761:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668972997760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828640.439767:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5090 00000100:00080000:3.0:1787828640.439775:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.439783:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828640.439784:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828640.439789:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828640.439793:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828640.439800:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972997760:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828640.439806:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668972998272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828640.439832:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.439835:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828640.439838:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828640.439842:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5090 00000100:00100000:1.0:1787828640.439847:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972997888:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828640.439849:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.439859:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828640.439865:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828640.439880:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668972998400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828640.439892:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828640.439896:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828640.439898:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828640.439903:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828640.439906:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5090 00000100:00080000:3.0:1787828640.439913:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828640.439918:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828640.439923:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828640.439926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828640] req@ffff9c1a7895cb40 x1874668972997888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828640.439939:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668972998528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828640.439948:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668972997888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828640.439963:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828640.439968:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828640.439972:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972997888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828640.439980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972998016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828640.440003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828640.440011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828640] req@ffff9c1a7895f0c0 x1874668972998016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828640.440024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668972998016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828640.440032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828640.440036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828640.440039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972998016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828640.440046:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972998144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828640.440067:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828640.440079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828640] req@ffff9c1a7895f840 x1874668972998144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828640.440090:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668972998144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828640.440106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828640.440110:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828640.440114:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972998144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828640.440121:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972998272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828640.440142:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828640.440154:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828640] req@ffff9c1a7895fc00 x1874668972998272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828640.440167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668972998272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828640.440178:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828640.440181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828640.440185:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972998272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828640.440206:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972998400:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828640.440228:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828640.440240:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828640] req@ffff9c1a7895e1c0 x1874668972998400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828640.440253:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668972998400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828640.440262:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828640.440265:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828640.440270:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972998400:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828640.440276:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972998528:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828640.440296:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828640.440325:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828640/real 1787828640] req@ffff9c1a7895de00 x1874668972998528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828640.440339:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668972998528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828640.440371:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828640.440375:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828640.440379:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972998528:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828640.951497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828640.951498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828641.207636:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828641.975418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828641.975469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828642.231577:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828642.999456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828642.999472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828643.255525:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828644.023424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828644.023446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828644.279461:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828645.047497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828645.047499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828645.303567:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828645.559471:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828645.559489:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828645.559497:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828645.559530:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5095 00000100:00080000:0.0:1787828645.559556:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828645.559567:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828645.559660:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668972998656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828645.559691:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828645.559695:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828645.559697:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828645.559701:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5095 00000100:00080000:0.0:1787828645.559706:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828645.559710:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828645.559723:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828645.559752:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668972998784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828645.559767:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972998656:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828645.559768:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828645.559780:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828645.559784:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828645.559790:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5095 00000100:00080000:0.0:1787828645.559796:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828645.559802:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828645.559807:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828645.559823:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668972998912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828645.559834:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828645.559838:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828645.559840:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828645.559843:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5095 00000100:00080000:0.0:1787828645.559846:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828645.559852:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828645.559853:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828645.559861:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828645.559872:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668972999040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828645.559879:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828645.559882:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828645.559883:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828645.559886:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5095 00000100:00080000:0.0:1787828645.559889:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828645.559893:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828645.559896:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828645.559928:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828645/real 1787828645] req@ffff9c1a788621c0 x1874668972998656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828645.559929:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972999168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828645.559965:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828645.559969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668972998656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828645.559971:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828645.559975:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828645.559987:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5095 00000100:00080000:0.0:1787828645.559993:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828645.559998:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828645.560013:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828645.560017:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828645.560024:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828645.560033:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972998656:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828645.560036:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972999296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828645.560054:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828645.560059:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828645.560062:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828645.560067:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972998784:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828645.560074:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5095 00000100:00080000:0.0:1787828645.560080:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828645.560084:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828645.560090:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828645.560103:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828645.560108:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972999424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828645.560140:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828645/real 1787828645] req@ffff9c1a78860780 x1874668972998784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828645.560155:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668972998784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828645.560162:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828645.560165:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828645.560168:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972998784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828645.560173:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972998912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828645.560209:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828645.560218:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828645/real 1787828645] req@ffff9c1a78861e00 x1874668972998912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828645.560228:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668972998912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828645.560236:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828645.560239:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828645.560241:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972998912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828645.560248:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972999040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828645.560262:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828645.560289:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828645/real 1787828645] req@ffff9c1a78862580 x1874668972999040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828645.560319:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668972999040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828645.560326:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828645.560328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828645.560331:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972999040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828645.560336:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972999168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828645.560354:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828645.560402:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828645/real 1787828645] req@ffff9c1a78861680 x1874668972999168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828645.560418:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972999168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828645.560428:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828645.560432:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828645.560436:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972999168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828645.560443:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972999296:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828645.560470:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828645.560482:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828645/real 1787828645] req@ffff9c1a788630c0 x1874668972999296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828645.560496:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972999296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828645.560505:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828645.560509:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828645.560512:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972999296:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828645.560557:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972999424:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828645.560572:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828645.560580:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828645/real 1787828645] req@ffff9c1a78863c00 x1874668972999424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828645.560589:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972999424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828645.560596:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828645.560598:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828645.560601:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972999424:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828646.071432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828646.071432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828646.327542:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828647.095533:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828647.095533:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828647.351564:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828648.119450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828648.119450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828648.375565:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828649.143503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828649.143503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828649.399585:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828650.167424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828650.167425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828650.423566:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828650.679424:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828650.679434:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828650.679439:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828650.679449:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5101 00000100:00080000:0.0:1787828650.679460:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828650.679467:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828650.679504:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668972999552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828650.679534:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828650.679538:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828650.679539:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828650.679544:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5101 00000100:00080000:0.0:1787828650.679547:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828650.679551:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828650.679560:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828650.679573:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668972999680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828650.679584:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828650.679588:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828650.679590:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828650.679595:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5101 00000100:00080000:0.0:1787828650.679599:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828650.679603:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828650.679608:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828650.679620:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668972999808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828650.679646:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972999552:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828650.679653:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828650.679670:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828650.679672:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828650.679679:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5101 00000100:00080000:0.0:1787828650.679684:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828650.679689:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828650.679697:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828650.679716:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668972999936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828650.679725:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828650.679729:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828650.679731:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828650.679734:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5101 00000100:00080000:0.0:1787828650.679741:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828650.679747:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828650.679753:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828650.679760:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828650.679768:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973000064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828650.679785:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828650.679789:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828650.679791:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828650.679796:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5101 00000100:00080000:0.0:1787828650.679802:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828650.679808:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828650.679817:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828650.679829:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973000192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828650.679841:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828650.679848:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828650/real 1787828650] req@ffff9c1a78860000 x1874668972999552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828650.679849:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828650.679854:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828650.679866:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5101 00000100:00080000:0.0:1787828650.679879:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828650.679890:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828650.679897:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828650.679908:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668972999552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828650.679916:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973000320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828650.679944:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828650.679950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828650.679957:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668972999552:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828650.679970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972999680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828650.680003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828650.680017:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828650/real 1787828650] req@ffff9c1a78863c00 x1874668972999680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828650.680030:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668972999680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828650.680038:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828650.680042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828650.680045:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972999680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828650.680076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972999808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828650.680093:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828650.680102:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828650/real 1787828650] req@ffff9c1a788630c0 x1874668972999808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828650.680121:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668972999808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828650.680129:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828650.680131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828650.680134:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668972999808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828650.680140:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972999936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828650.680154:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828650.680163:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828650/real 1787828650] req@ffff9c1a78861680 x1874668972999936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828650.680174:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668972999936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828650.680182:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828650.680185:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828650.680187:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668972999936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828650.680193:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973000064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828650.680207:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828650.680216:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828650/real 1787828650] req@ffff9c1a78862580 x1874668973000064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828650.680226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973000064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828650.680263:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828650.680266:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828650.680270:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973000064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828650.680276:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973000192:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828650.680300:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828650.680311:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828650/real 1787828650] req@ffff9c1a78861e00 x1874668973000192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828650.680324:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973000192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828650.680336:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828650.680341:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828650.680426:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973000192:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828650.680436:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973000320:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828650.680480:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828650.680493:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828650/real 1787828650] req@ffff9c1a78860780 x1874668973000320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828650.680507:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973000320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828650.680520:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828650.680525:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828650.680531:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973000320:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828651.191496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828651.191496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828651.447455:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828652.215428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828652.215457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828652.471472:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828653.239441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828653.239442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828653.495476:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828654.263394:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828654.263408:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828654.519545:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828655.287409:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828655.287413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828655.543527:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828655.799474:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828655.799483:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828655.799487:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828655.799495:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5106 00000100:00080000:0.0:1787828655.799505:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828655.799511:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828655.799545:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973000448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828655.799562:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828655.799565:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828655.799567:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828655.799570:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5106 00000100:00080000:0.0:1787828655.799573:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828655.799576:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828655.799584:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828655.799591:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973000576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828655.799601:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973000448:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828655.799602:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828655.799607:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828655.799609:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828655.799615:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5106 00000100:00080000:0.0:1787828655.799618:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828655.799620:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828655.799622:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828655.799630:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973000704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828655.799637:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828655.799639:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828655.799640:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828655.799643:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5106 00000100:00080000:0.0:1787828655.799646:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828655.799650:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828655.799655:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828655.799668:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973000832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828655.799678:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828655.799682:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828655.799685:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828655.799689:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5106 00000100:00080000:0.0:1787828655.799693:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828655.799700:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787828655.799703:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787828655.799707:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828655.799723:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973000960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828655.799731:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828655.799733:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828655.799735:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828655.799737:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5106 00000100:00080000:0.0:1787828655.799741:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828655.799744:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828655.799748:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828655.799756:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973001088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828655.799762:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828655.799766:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828655.799766:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828655/real 1787828655] req@ffff9c1a788621c0 x1874668973000448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828655.799768:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828655.799783:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5106 00000100:00080000:0.0:1787828655.799786:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828655.799789:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828655.799793:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828655.799799:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973000448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828655.799803:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973001216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828655.799831:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828655.799840:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828655.799845:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973000448:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828655.799871:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973000576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828655.799898:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828655.799914:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828655/real 1787828655] req@ffff9c1a78862940 x1874668973000576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828655.799928:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973000576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828655.799948:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828655.799953:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828655.799962:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973000576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828655.799970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973000704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828655.799994:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828655.800008:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828655/real 1787828655] req@ffff9c1a78861a40 x1874668973000704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828655.800022:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973000704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828655.800030:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828655.800035:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828655.800039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973000704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828655.800054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973000832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828655.800074:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828655.800086:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828655/real 1787828655] req@ffff9c1a78860f00 x1874668973000832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828655.800098:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973000832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828655.800107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828655.800110:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828655.800114:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973000832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828655.800119:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973000960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828655.800139:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828655.800150:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828655/real 1787828655] req@ffff9c1a78862d00 x1874668973000960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828655.800163:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973000960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828655.800172:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828655.800175:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828655.800179:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973000960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828655.800185:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973001088:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828655.800221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828655.800233:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828655/real 1787828655] req@ffff9c1a78860b40 x1874668973001088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828655.800245:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973001088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828655.800255:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828655.800258:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828655.800261:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973001088:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828655.800268:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973001216:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828655.800288:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828655.800299:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828655/real 1787828655] req@ffff9c1a78863840 x1874668973001216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828655.800320:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973001216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828655.800328:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828655.800331:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828655.800335:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973001216:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828656.311420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828656.311430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828656.567513:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828657.335443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828657.335443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828657.591551:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828658.359478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828658.359512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828658.615431:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828659.383426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828659.383444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828659.639554:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828660.407453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828660.407453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828660.663522:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828660.919443:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828660.919452:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828660.919456:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828660.919466:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5111 00000100:00080000:0.0:1787828660.919475:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828660.919482:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828660.919514:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973001344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828660.919536:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828660.919539:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828660.919541:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828660.919544:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5111 00000100:00080000:0.0:1787828660.919547:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828660.919549:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828660.919556:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828660.919567:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973001472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828660.919581:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828660.919584:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828660.919585:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828660.919588:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5111 00000100:00080000:0.0:1787828660.919590:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828660.919592:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828660.919596:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828660.919600:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973001344:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828660.919606:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973001600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828660.919625:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828660.919627:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828660.919628:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828660.919630:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5111 00000100:00080000:0.0:1787828660.919633:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828660.919637:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828660.919640:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828660.919650:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973001728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828660.919658:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828660.919660:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828660.919662:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828660.919664:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5111 00000100:00080000:0.0:1787828660.919670:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828660.919673:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828660.919676:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828660.919688:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973001856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828660.919690:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828660.919699:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828660.919702:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828660.919704:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828660.919707:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5111 00000100:00080000:0.0:1787828660.919710:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828660.919713:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828660.919716:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828660.919724:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973001984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828660.919731:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828660.919734:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828660.919735:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828660.919740:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5111 00000100:00100000:1.0:1787828660.919742:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828660/real 1787828660] req@ffff9c1a788612c0 x1874668973001344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828660.919743:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828660.919750:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828660.919756:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828660.919767:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973002112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828660.919778:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973001344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828660.919794:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828660.919801:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828660.919807:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973001344:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828660.919818:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973001472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828660.919846:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828660.919861:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828660/real 1787828660] req@ffff9c1a78863840 x1874668973001472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828660.919876:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973001472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828660.919886:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828660.919890:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828660.919894:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973001472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828660.919921:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973001600:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828660.919943:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828660.919964:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828660/real 1787828660] req@ffff9c1a78860b40 x1874668973001600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828660.919979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973001600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828660.919989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828660.919992:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828660.919996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973001600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828660.920002:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973001728:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828660.920024:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828660.920036:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828660/real 1787828660] req@ffff9c1a78862d00 x1874668973001728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828660.920056:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973001728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828660.920066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828660.920070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828660.920074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973001728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828660.920080:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973001856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828660.920099:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828660.920112:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828660/real 1787828660] req@ffff9c1a78860f00 x1874668973001856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828660.920125:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973001856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828660.920134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828660.920138:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828660.920141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973001856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828660.920148:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973001984:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828660.920187:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828660.920200:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828660/real 1787828660] req@ffff9c1a78861a40 x1874668973001984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828660.920213:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973001984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828660.920229:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828660.920233:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828660.920237:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973001984:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828660.920244:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973002112:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828660.920264:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828660.920276:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828660/real 1787828660] req@ffff9c1a78862940 x1874668973002112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828660.920289:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973002112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828660.920299:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828660.920302:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828660.920306:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973002112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828661.431436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828661.431445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828661.687506:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828662.455459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828662.455459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828662.711512:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828663.479475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828663.479483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828663.735560:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828664.503465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828664.503465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828664.759527:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828665.527425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828665.527425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828665.783482:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828666.039476:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828666.039485:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828666.039489:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828666.039498:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5116 00000100:00080000:2.0:1787828666.039508:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828666.039514:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828666.039547:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973002240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828666.039571:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828666.039574:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828666.039576:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828666.039580:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5116 00000100:00080000:2.0:1787828666.039583:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828666.039585:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828666.039593:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828666.039602:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973002368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828666.039610:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828666.039613:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828666.039619:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828666.039621:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5116 00000100:00080000:2.0:1787828666.039624:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828666.039626:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828666.039630:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828666.039637:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973002496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828666.039646:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828666.039649:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828666.039650:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828666.039653:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5116 00000100:00080000:2.0:1787828666.039655:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828666.039658:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828666.039661:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828666.039672:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973002624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828666.039673:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973002240:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828666.039688:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828666.039691:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828666.039693:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828666.039696:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5116 00000100:00080000:2.0:1787828666.039699:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828666.039701:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828666.039705:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828666.039713:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973002752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828666.039721:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828666.039723:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828666.039724:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828666.039726:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5116 00000100:00080000:2.0:1787828666.039730:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828666.039732:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828666.039736:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828666.039746:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828666.039747:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973002880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828666.039757:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828666.039761:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828666.039764:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828666.039769:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5116 00000100:00080000:2.0:1787828666.039774:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828666.039778:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828666.039785:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828666.039786:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff9c1a49d8d680 x1874668973002240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828666.039805:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973003008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828666.039812:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973002240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.039836:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.039841:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828666.039845:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973002240:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828666.039851:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973002368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828666.039869:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828666.039877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff9c1a49d8ed00 x1874668973002368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828666.039886:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973002368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.039893:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.039895:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828666.039897:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973002368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828666.039901:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973002496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828666.039912:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828666.039937:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff9c1a49d8fc00 x1874668973002496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828666.039946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973002496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.039958:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.039961:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828666.039964:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973002496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828666.039988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973002624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828666.040002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828666.040009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff9c1a49d8e580 x1874668973002624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828666.040018:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973002624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.040025:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.040028:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828666.040030:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973002624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828666.040035:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973002752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828666.040047:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828666.040053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff9c1a49d8c3c0 x1874668973002752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828666.040062:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973002752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.040069:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.040071:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828666.040074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973002752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828666.040079:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973002880:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828666.040092:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828666.040098:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff9c1a49d8e1c0 x1874668973002880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828666.040107:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973002880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.040113:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.040115:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828666.040118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973002880:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828666.040122:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973003008:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828666.040153:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828666.040162:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828666/real 1787828666] req@ffff9c1a49d8f0c0 x1874668973003008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828666.040190:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973003008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828666.040196:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828666.040199:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828666.040201:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973003008:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828666.551479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828666.551479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828666.807537:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828667.575434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828667.575434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828667.831476:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828668.599448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828668.599448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828668.855518:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828669.623426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828669.623446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828669.879535:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828670.647450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828670.647450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828670.903536:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828671.159453:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828671.159481:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828671.159486:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828671.159498:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5121 00000100:00080000:2.0:1787828671.159509:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828671.159518:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828671.159556:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973003136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828671.159592:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828671.159596:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828671.159599:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828671.159604:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5121 00000100:00080000:2.0:1787828671.159610:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828671.159615:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828671.159626:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828671.159643:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973003136:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828671.159644:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973003264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828671.159660:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828671.159664:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828671.159667:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828671.159672:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5121 00000100:00080000:2.0:1787828671.159677:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828671.159682:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828671.159689:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828671.159704:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973003392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828671.159717:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828671.159733:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828671.159736:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828671.159741:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5121 00000100:00080000:2.0:1787828671.159747:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828671.159752:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828671.159757:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828671.159771:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973003520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828671.159775:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828671.159787:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828671.159790:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828671.159792:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828671.159798:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5121 00000100:00080000:2.0:1787828671.159802:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828671.159807:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828671.159813:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828671.159814:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff9c1a49d8de00 x1874668973003136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828671.159828:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973003648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828671.159843:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828671.159846:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828671.159848:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828671.159851:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5121 00000100:00080000:2.0:1787828671.159855:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828671.159858:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828671.159863:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828671.159873:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973003776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828671.159882:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828671.159885:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828671.159887:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828671.159891:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5121 00000100:00080000:2.0:1787828671.159895:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828671.159899:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828671.159904:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828671.159917:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973003904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828671.159953:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973003136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828671.159963:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828671.159967:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828671.159970:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973003136:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828671.160024:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973003264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828671.160043:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828671.160053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff9c1a49d8c780 x1874668973003264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828671.160062:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973003264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828671.160073:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828671.160075:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828671.160077:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973003264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828671.160081:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973003392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828671.160105:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828671.160115:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff9c1a49d8c000 x1874668973003392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828671.160123:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973003392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828671.160129:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828671.160131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828671.160133:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973003392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828671.160137:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973003520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828671.160152:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828671.160158:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff9c1a49d8cf00 x1874668973003520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828671.160166:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973003520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828671.160171:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828671.160173:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828671.160175:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973003520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828671.160179:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973003648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828671.160190:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828671.160196:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff9c1a49d8da40 x1874668973003648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828671.160202:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973003648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828671.160208:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828671.160210:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828671.160212:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973003648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828671.160215:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973003776:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828671.160226:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828671.160231:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff9c1a49d8cb40 x1874668973003776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828671.160238:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973003776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828671.160244:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828671.160245:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828671.160247:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973003776:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828671.160250:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973003904:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828671.160260:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828671.160266:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828671/real 1787828671] req@ffff9c1a49d8f0c0 x1874668973003904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828671.160273:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973003904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828671.160278:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828671.160280:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828671.160281:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973003904:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828671.671413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828671.671419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828671.927518:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828672.695454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828672.695454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828672.951547:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828673.719418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828673.719422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828673.975472:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828674.743438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828674.743438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828674.999531:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828675.767453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828675.767453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828676.023539:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828676.279467:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828676.279475:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828676.279479:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828676.279495:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5126 00000100:00080000:0.0:1787828676.279504:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828676.279509:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828676.279541:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973004032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828676.279573:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828676.279576:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828676.279578:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828676.279586:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5126 00000100:00080000:0.0:1787828676.279589:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828676.279593:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828676.279601:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828676.279611:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973004160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828676.279617:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973004032:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828676.279622:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828676.279626:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828676.279628:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828676.279631:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5126 00000100:00080000:0.0:1787828676.279634:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828676.279636:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828676.279640:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828676.279649:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973004288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828676.279657:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828676.279660:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828676.279661:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828676.279664:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5126 00000100:00080000:0.0:1787828676.279666:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828676.279668:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828676.279671:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828676.279679:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973004416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828676.279688:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828676.279688:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828676.279694:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828676.279695:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828676.279698:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5126 00000100:00080000:0.0:1787828676.279700:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828676.279703:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828676.279705:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828676.279712:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973004544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828676.279723:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828676.279725:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828676.279727:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828676.279730:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5126 00000100:00080000:0.0:1787828676.279733:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828676.279735:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff9c1a788621c0 x1874668973004032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828676.279737:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828676.279742:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828676.279753:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973004672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828676.279759:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828676.279763:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828676.279765:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828676.279765:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973004032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828676.279771:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5126 00000100:00080000:0.0:1787828676.279777:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828676.279782:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828676.279789:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828676.279791:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828676.279797:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973004800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828676.279801:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828676.279811:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973004032:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828676.279836:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973004160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828676.279863:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828676.279877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff9c1a78860780 x1874668973004160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828676.279897:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973004160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828676.279906:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828676.279910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828676.279913:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973004160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828676.279920:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973004288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828676.279939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828676.279950:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff9c1a78861e00 x1874668973004288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828676.279962:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973004288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828676.279970:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828676.279974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828676.279977:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973004288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828676.279984:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973004416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828676.280000:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828676.280010:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff9c1a78862580 x1874668973004416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828676.280022:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973004416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828676.280037:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828676.280041:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828676.280044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973004416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828676.280050:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973004544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828676.280083:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828676.280094:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff9c1a78861680 x1874668973004544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828676.280115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973004544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828676.280124:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828676.280127:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828676.280131:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973004544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828676.280137:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973004672:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828676.280155:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828676.280171:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff9c1a788630c0 x1874668973004672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828676.280184:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973004672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828676.280192:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828676.280195:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828676.280199:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973004672:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828676.280205:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973004800:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828676.280222:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828676.280232:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828676/real 1787828676] req@ffff9c1a78863c00 x1874668973004800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828676.280244:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973004800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828676.280252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828676.280255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828676.280259:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973004800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828676.791458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828676.791458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828677.047508:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828677.815482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828677.815488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828678.071567:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828678.839407:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828678.839424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828679.095511:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828679.863455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828679.863455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828680.119474:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828680.887413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828680.887428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828681.143491:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828681.399459:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828681.399470:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828681.399475:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828681.399485:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5131 00000100:00080000:0.0:1787828681.399497:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828681.399505:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828681.399543:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973004928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828681.399582:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828681.399587:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828681.399589:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828681.399594:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5131 00000100:00080000:0.0:1787828681.399598:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828681.399603:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828681.399613:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828681.399634:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973005056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828681.399635:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973004928:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828681.399655:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828681.399659:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828681.399661:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828681.399665:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5131 00000100:00080000:0.0:1787828681.399670:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828681.399675:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828681.399681:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828681.399694:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973005184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828681.399699:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828681.399713:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828681.399719:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828681.399721:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828681.399726:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5131 00000100:00080000:0.0:1787828681.399731:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828681.399735:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff9c1a78860000 x1874668973004928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828681.399740:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828681.399751:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828681.399757:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973004928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828681.399767:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973005312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828681.399779:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828681.399786:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828681.399789:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828681.399791:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973004928:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828681.399799:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828681.399803:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828681.399809:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5131 00000100:00080000:0.0:1787828681.399813:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828681.399819:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973005056:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828681.399821:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828681.399830:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828681.399844:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828681.399848:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973005440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828681.399858:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff9c1a78863c00 x1874668973005056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828681.399868:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828681.399873:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973005056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828681.399877:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828681.399882:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828681.399885:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828681.399891:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5131 00000100:00080000:3.0:1787828681.399891:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828681.399912:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973005056:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828681.399916:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828681.399919:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973005184:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828681.399926:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828681.399934:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828681.399939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828681.399949:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff9c1a788630c0 x1874668973005184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828681.399954:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973005568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828681.399963:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973005184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828681.399976:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828681.399977:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828681.399983:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828681.399985:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828681.399988:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973005184:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828681.399990:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828681.399994:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973005312:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828681.400000:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5131 00000100:00080000:0.0:1787828681.400006:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828681.400011:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828681.400014:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828681.400037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff9c1a78861680 x1874668973005312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828681.400039:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828681.400072:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973005312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828681.400077:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973005696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828681.400082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828681.400088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828681.400097:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973005312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828681.400119:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973005440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828681.400143:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828681.400154:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff9c1a78862580 x1874668973005440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828681.400164:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973005440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828681.400170:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828681.400173:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828681.400176:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973005440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828681.400182:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973005568:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828681.400195:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828681.400201:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff9c1a78861e00 x1874668973005568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828681.400208:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973005568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828681.400214:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828681.400216:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828681.400218:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973005568:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828681.400222:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973005696:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828681.400231:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828681.400236:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828681/real 1787828681] req@ffff9c1a78860780 x1874668973005696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828681.400242:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973005696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828681.400248:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828681.400249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828681.400251:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973005696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828681.911431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828681.911431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828682.167517:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828682.935413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828682.935413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828683.191531:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828683.959486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828683.959495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828684.215652:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828684.983489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828684.983489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828685.239516:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828686.007494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828686.007494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828686.263498:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828686.519505:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828686.519517:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828686.519521:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828686.519532:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5136 00000100:00080000:2.0:1787828686.519543:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828686.519551:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828686.519589:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973005824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828686.519613:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828686.519616:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828686.519628:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828686.519632:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5136 00000100:00080000:2.0:1787828686.519635:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828686.519638:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828686.519646:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828686.519655:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973005952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828686.519663:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828686.519666:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828686.519683:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828686.519687:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5136 00000100:00080000:2.0:1787828686.519690:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828686.519694:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828686.519698:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828686.519710:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973006080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828686.519721:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973005824:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828686.519729:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828686.519751:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828686.519753:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828686.519760:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5136 00000100:00080000:2.0:1787828686.519765:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828686.519771:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828686.519778:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828686.519793:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973006208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828686.519802:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828686.519807:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828686.519808:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828686.519811:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828686.519827:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5136 00000100:00080000:2.0:1787828686.519849:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828686.519853:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828686.519859:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828686.519870:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973006336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828686.519889:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828686.519890:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828686/real 1787828686] req@ffff9c1a49d8e1c0 x1874668973005824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828686.519914:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828686.519917:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828686.519928:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5136 00000100:00080000:2.0:1787828686.519932:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828686.519937:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828686.519942:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828686.519943:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973005824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828686.519957:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973006464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828686.519958:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828686.519969:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787828686.519979:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828686.519980:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973005824:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828686.520006:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828686.520008:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828686.520017:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5136 00000100:00100000:3.0:1787828686.520022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973005952:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828686.520024:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828686.520056:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828686.520062:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828686.520076:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973006592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828686.520083:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828686.520102:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828686/real 1787828686] req@ffff9c1a49d8c3c0 x1874668973005952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828686.520115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973005952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828686.520123:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828686.520127:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828686.520130:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973005952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828686.520153:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973006080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828686.520166:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828686.520174:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828686/real 1787828686] req@ffff9c1a49d8e580 x1874668973006080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828686.520183:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973006080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828686.520188:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828686.520191:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828686.520193:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973006080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828686.520197:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973006208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828686.520209:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828686.520215:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828686/real 1787828686] req@ffff9c1a49d8fc00 x1874668973006208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828686.520247:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973006208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828686.520254:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828686.520256:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828686.520258:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973006208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828686.520263:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973006336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828686.520277:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828686.520285:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828686/real 1787828686] req@ffff9c1a49d8ed00 x1874668973006336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828686.520310:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973006336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828686.520317:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828686.520319:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828686.520321:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973006336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828686.520326:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973006464:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828686.520341:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828686.520348:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828686/real 1787828686] req@ffff9c1a49d8d680 x1874668973006464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828686.520357:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973006464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828686.520387:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828686.520390:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828686.520392:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973006464:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828686.520397:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973006592:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828686.520463:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828686.520472:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828686/real 1787828686] req@ffff9c1a49d8f0c0 x1874668973006592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828686.520480:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973006592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828686.520486:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828686.520488:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828686.520490:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973006592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828687.031408:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828687.031421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828687.287493:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828688.055423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828688.055424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828688.311657:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828689.079513:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828689.079519:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828689.335541:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828690.103517:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828690.103523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828690.359564:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828691.127416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828691.127418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828691.383505:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828691.639483:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828691.639491:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828691.639495:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828691.639504:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5141 00000100:00080000:0.0:1787828691.639514:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828691.639520:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828691.639551:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973006720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828691.639574:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828691.639584:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828691.639586:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828691.639589:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5141 00000100:00080000:0.0:1787828691.639592:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828691.639595:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828691.639603:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828691.639615:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973006848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828691.639617:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973006720:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828691.639631:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828691.639634:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828691.639635:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828691.639638:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5141 00000100:00080000:0.0:1787828691.639640:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828691.639642:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828691.639645:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828691.639654:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973006976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828691.639661:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828691.639664:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828691.639665:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828691.639668:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5141 00000100:00080000:0.0:1787828691.639670:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828691.639672:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828691.639675:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828691.639682:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973007104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828691.639688:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828691.639691:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828691.639697:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828691.639699:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828691.639703:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5141 00000100:00080000:0.0:1787828691.639706:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828691.639709:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828691.639712:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828691.639721:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973007232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828691.639728:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828691.639728:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828691/real 1787828691] req@ffff9c1a788621c0 x1874668973006720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828691.639737:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828691.639739:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828691.639747:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5141 00000100:00080000:0.0:1787828691.639751:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828691.639753:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828691.639757:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828691.639758:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973006720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828691.639768:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973007360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828691.639770:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828691.639779:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828691.639781:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828691.639785:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973006720:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828691.639786:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828691.639790:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828691.639795:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5141 00000100:00080000:0.0:1787828691.639796:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828691.639799:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828691.639801:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828691.639808:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973007488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828691.639811:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973006848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828691.639837:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828691.639848:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828691/real 1787828691] req@ffff9c1a78862940 x1874668973006848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828691.639856:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973006848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828691.639862:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828691.639864:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828691.639866:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973006848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828691.639870:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973006976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828691.639880:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828691.639886:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828691/real 1787828691] req@ffff9c1a78861a40 x1874668973006976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828691.639893:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973006976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828691.639898:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828691.639900:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828691.639902:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973006976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828691.639905:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973007104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828691.639915:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828691.639921:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828691/real 1787828691] req@ffff9c1a78860f00 x1874668973007104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828691.639929:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973007104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828691.639934:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828691.639935:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828691.639937:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973007104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828691.639941:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973007232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828691.639963:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828691.639973:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828691/real 1787828691] req@ffff9c1a78862d00 x1874668973007232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828691.639981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973007232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828691.639986:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828691.639989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828691.639991:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973007232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828691.639994:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973007360:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828691.640008:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828691.640014:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828691/real 1787828691] req@ffff9c1a78860b40 x1874668973007360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828691.640026:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973007360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828691.640031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828691.640033:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828691.640034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973007360:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828691.640046:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973007488:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828691.640057:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828691.640062:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828691/real 1787828691] req@ffff9c1a78863840 x1874668973007488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828691.640070:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973007488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828691.640075:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828691.640077:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828691.640079:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973007488:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828692.151418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828692.151418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828692.407543:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828693.175498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828693.175508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828693.431554:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828694.199425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828694.199425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828694.455458:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828695.223543:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828695.223543:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828695.479648:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828696.247446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828696.247448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828696.503553:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828696.759512:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828696.759523:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828696.759528:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828696.759539:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5147 00000100:00080000:0.0:1787828696.759551:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828696.759560:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828696.759598:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973007616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828696.759623:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828696.759627:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828696.759629:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828696.759633:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5147 00000100:00080000:0.0:1787828696.759638:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828696.759642:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828696.759652:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828696.759663:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973007616:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828696.759670:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973007744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828696.759712:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828696.759715:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828696.759717:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828696.759721:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5147 00000100:00080000:0.0:1787828696.759726:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828696.759732:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828696.759740:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828696.759758:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973007872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828696.759763:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828696.759780:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828696.759786:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828696.759789:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828696.759796:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5147 00000100:00080000:0.0:1787828696.759802:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828696.759809:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828696/real 1787828696] req@ffff9c1a788612c0 x1874668973007616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828696.759811:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828696.759825:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828696.759857:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973007616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828696.759871:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973008000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828696.759881:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828696.759894:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828696.759895:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828696.759908:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787828696.759910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973007616:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828696.759913:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828696.759965:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5147 00000100:00080000:0.0:1787828696.759970:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828696.759974:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828696.759980:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828696.759987:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973007744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828696.759996:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973008128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828696.760030:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828696.760035:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828696.760038:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828696.760051:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5147 00000800:00000100:3.0:1787828696.760051:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828696.760065:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828696.760071:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828696.760080:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828696.760088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828696/real 1787828696] req@ffff9c1a78863840 x1874668973007744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828696.760101:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973008256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828696.760134:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828696.760139:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828696.760143:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828696.760149:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973007744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828696.760151:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5147 00000100:00080000:0.0:1787828696.760160:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828696.760170:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828696.760183:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828696.760200:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828696.760203:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973008384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828696.760213:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828696.760241:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973007744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828696.760251:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973007872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828696.760276:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828696.760286:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828696/real 1787828696] req@ffff9c1a78860b40 x1874668973007872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828696.760296:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973007872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828696.760304:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828696.760324:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828696.760328:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973007872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828696.760337:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973008000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828696.760418:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828696.760431:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828696/real 1787828696] req@ffff9c1a78862d00 x1874668973008000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828696.760462:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973008000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828696.760479:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828696.760483:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828696.760487:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973008000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828696.760509:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973008128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828696.760535:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828696.760547:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828696/real 1787828696] req@ffff9c1a78860f00 x1874668973008128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828696.760575:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973008128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828696.760593:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828696.760595:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828696.760598:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973008128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828696.760602:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973008256:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828696.760635:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828696.760643:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828696/real 1787828696] req@ffff9c1a78861a40 x1874668973008256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828696.760653:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973008256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828696.760660:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828696.760662:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828696.760664:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973008256:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828696.760669:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973008384:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828696.760682:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828696.760689:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828696/real 1787828696] req@ffff9c1a78862940 x1874668973008384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828696.760698:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973008384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828696.760705:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828696.760707:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828696.760709:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973008384:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828697.271432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828697.271471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828697.527478:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828698.295456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828698.295460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828698.551557:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828699.319500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828699.319501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828699.575592:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828700.343458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828700.343471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828700.599492:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828701.367506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828701.367508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828701.623495:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828701.879508:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828701.879519:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828701.879524:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828701.879535:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5152 00000100:00080000:1.0:1787828701.879548:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828701.879556:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828701.879602:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973008512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828701.879624:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828701.879627:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828701.879629:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828701.879632:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5152 00000100:00080000:1.0:1787828701.879634:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828701.879637:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828701.879648:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828701.879656:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973008640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828701.879662:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828701.879665:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828701.879666:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828701.879669:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5152 00000100:00080000:1.0:1787828701.879671:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828701.879673:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828701.879676:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828701.879682:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973008768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828701.879690:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828701.879693:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828701.879696:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828701.879698:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973008512:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828701.879701:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5152 00000100:00080000:1.0:1787828701.879707:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828701.879709:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828701.879712:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828701.879720:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973008896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828701.879726:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828701.879729:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828701.879730:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828701.879733:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5152 00000100:00080000:1.0:1787828701.879736:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828701.879738:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828701.879741:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828701.879749:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973009024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828701.879754:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828701.879756:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828701.879757:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828701.879761:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5152 00000100:00080000:1.0:1787828701.879764:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828701.879766:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828701.879770:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828701.879777:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973009152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828701.879783:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828701.879786:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828701.879787:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828701.879790:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5152 00000100:00080000:1.0:1787828701.879792:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828701.879794:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828701.879798:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828701.879806:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973009280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828701.879810:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973008640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828701.879848:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973008768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828701.879868:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973008896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828701.879902:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973009024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828701.879936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973009152:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828701.879953:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973009280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828702.391436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828702.391441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828702.647570:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828703.415493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828703.415493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828703.671557:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828704.439453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828704.439453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828704.695544:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828704.951606:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787828704.951654:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828704.951674:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828704.951722:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828704.951734:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828704.951748:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828704.951777:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787828704.951779:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828701/real 1787828704] req@ffff9c1a78a7bc00 x1874668973008768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828704.951794:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787828704.951822:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973008768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828704.951828:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00080000:3.0:1787828704.951848:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828704.951856:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828704.951879:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973008768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828704.951888:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828701/real 1787828704] req@ffff9c1a78a78780 x1874668973008896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828704.951916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973008896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828704.951926:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828704.951929:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828704.951933:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973008896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828704.951941:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828701/real 1787828704] req@ffff9c1a78a7a1c0 x1874668973009024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828704.951961:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973009024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828704.951981:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828704.951984:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828704.951988:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973009024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828704.951995:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828701/real 1787828704] req@ffff9c1a78a783c0 x1874668973009152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828704.952007:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973009152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828704.952027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828704.952031:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828704.952035:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973009152:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828704.952043:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828701/real 1787828704] req@ffff9c1a78a7a940 x1874668973009280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828704.952057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973009280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828704.952068:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828704.952072:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828704.952077:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973009280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828704.952086:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828701/real 1787828704] req@ffff9c1a78a79e00 x1874668973008512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828704.952103:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973008512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828704.952113:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828704.952134:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828704.952138:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973008512:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828704.952147:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828701/real 1787828704] req@ffff9c1a78a7b0c0 x1874668973008640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828704.952161:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973008640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828704.952171:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828704.952175:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828704.952179:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973008640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828705.463427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828705.463427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828705.719562:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828706.487499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828706.487504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828706.743504:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828706.999492:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828706.999506:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828706.999512:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828706.999525:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5157 00000100:00080000:1.0:1787828706.999539:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828706.999551:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828706.999599:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973009408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828706.999643:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828706.999648:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828706.999651:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828706.999658:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5157 00000100:00080000:1.0:1787828706.999664:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828706.999669:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828706.999683:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828706.999694:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973009408:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828706.999719:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973009536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828706.999749:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828706.999756:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828706.999760:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828706.999769:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5157 00000100:00080000:1.0:1787828706.999778:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828706.999789:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787828706.999791:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:1.0:1787828706.999809:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828706.999843:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973009664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828706.999886:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828706/real 1787828706] req@ffff9c1a78a7b840 x1874668973009408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828706.999952:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828706.999960:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973009408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828706.999966:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828706.999972:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828706.999989:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5157 00000100:00080000:0.0:1787828706.999991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828707.000005:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828707.000007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828707.000024:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828707.000025:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973009408:192.168.203.148@tcp:250:kworker.0 00000008:00100000:1.0:1787828707.000050:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828707.000092:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973009792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828707.000108:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973009536:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828707.000131:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828707.000141:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828707.000147:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828707.000158:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5157 00000800:00000100:0.0:1787828707.000164:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828707.000175:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828707.000189:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828707.000199:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828706/real 1787828707] req@ffff9c1a78a792c0 x1874668973009536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828707.000204:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828707.000238:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973009536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828707.000250:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973009920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828707.000269:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828707.000286:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828707.000287:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828707.000304:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973009536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828707.000315:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973009664:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828707.000324:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828707.000329:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828707.000341:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5157 00000100:00080000:1.0:1787828707.000414:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828707.000424:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828707.000430:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828707.000446:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828707.000456:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828706/real 1787828707] req@ffff9c1a78a79a40 x1874668973009664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828707.000482:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973010048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828707.000497:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973009664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828707.000515:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828707.000520:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828707.000526:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828707.000534:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828707.000538:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828707.000550:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973009664:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828707.000559:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5157 00000100:00080000:1.0:1787828707.000573:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828707.000584:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828707.000596:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828707.000607:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973009792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828707.000628:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973010176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828707.000653:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828707.000679:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828707/real 1787828707] req@ffff9c1a78a7b480 x1874668973009792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828707.000698:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973009792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828707.000710:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828707.000717:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828707.000727:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973009792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828707.000738:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973009920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828707.000762:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828707.000776:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828707/real 1787828707] req@ffff9c1a78a78f00 x1874668973009920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828707.000789:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973009920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828707.000805:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828707.000808:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828707.000812:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973009920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828707.000819:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973010048:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828707.000843:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828707.000872:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828707/real 1787828707] req@ffff9c1a78a7a580 x1874668973010048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828707.000886:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973010048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828707.000895:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828707.000898:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828707.000902:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973010048:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828707.000924:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973010176:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828707.000947:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828707.000976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828707/real 1787828707] req@ffff9c1a78a78000 x1874668973010176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828707.000986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973010176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828707.000995:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828707.000999:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828707.001003:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973010176:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828707.511476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828707.511476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828707.767522:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828708.535414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828708.535422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828708.791460:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828709.559406:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828709.559424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828709.815538:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828710.583440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828710.583466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828710.839541:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828711.607452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828711.607477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828711.863550:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828712.119501:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828712.119511:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828712.119515:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.119525:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5162 00000100:00080000:2.0:1787828712.119544:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828712.119551:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828712.119586:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973010304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828712.119611:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828712.119614:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828712.119616:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.119620:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5162 00000100:00080000:2.0:1787828712.119623:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828712.119625:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828712.119633:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828712.119643:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973010432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828712.119652:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828712.119655:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828712.119657:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.119661:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5162 00000100:00080000:2.0:1787828712.119663:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828712.119665:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828712.119668:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828712.119679:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973010560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828712.119682:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973010304:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828712.119703:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828712.119707:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828712.119710:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.119715:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5162 00000100:00080000:2.0:1787828712.119719:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828712.119742:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828712.119751:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828712.119770:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973010688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828712.119785:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828712.119807:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828712.119824:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828712.119829:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.119836:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5162 00000100:00080000:2.0:1787828712.119842:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828712.119848:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828712.119853:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828712.119867:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973010816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828712.119876:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828712.119878:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff9c1a49d8cb40 x1874668973010304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828712.119879:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828712.119883:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828712.119892:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5162 00000100:00080000:2.0:1787828712.119898:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828712.119904:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828712.119908:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828712.119920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973010304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828712.119921:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973010944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828712.119942:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828712.119950:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828712.119952:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828712.119954:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828712.119955:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5162 00000100:00080000:2.0:1787828712.119961:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828712.119967:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787828712.119967:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828712.119977:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828712.119981:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973010304:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828712.119988:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973011072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828712.119999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973010432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828712.120035:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828712.120053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff9c1a49d8da40 x1874668973010432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828712.120068:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973010432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828712.120079:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828712.120083:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828712.120087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973010432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828712.120116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973010560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828712.120142:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828712.120156:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff9c1a49d8cf00 x1874668973010560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828712.120171:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973010560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828712.120182:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828712.120186:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828712.120190:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973010560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828712.120198:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973010688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828712.120221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828712.120235:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff9c1a49d8c000 x1874668973010688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828712.120251:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973010688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828712.120262:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828712.120266:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828712.120270:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973010688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828712.120277:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973010816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828712.120302:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828712.120317:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff9c1a49d8c780 x1874668973010816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828712.120332:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973010816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828712.120343:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828712.120380:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828712.120386:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973010816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828712.120394:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973010944:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828712.120421:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828712.120435:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff9c1a49d8de00 x1874668973010944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828712.120450:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973010944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828712.120461:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828712.120465:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828712.120470:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973010944:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828712.120477:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973011072:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828712.120502:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828712.120517:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828712/real 1787828712] req@ffff9c1a49d8f0c0 x1874668973011072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828712.120532:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973011072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828712.120542:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828712.120546:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828712.120551:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973011072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828712.631448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828712.631448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828712.887483:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828713.655427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828713.655427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828713.911535:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828714.679489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828714.679489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828714.935559:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828715.703431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828715.703431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828715.959450:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828716.727433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828716.727433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828716.983477:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828717.239415:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828717.239423:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828717.239427:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828717.239436:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5167 00000100:00080000:2.0:1787828717.239445:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828717.239452:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828717.239484:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973011200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828717.239508:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828717.239516:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828717.239518:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828717.239523:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5167 00000100:00080000:2.0:1787828717.239527:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828717.239533:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828717.239543:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828717.239554:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973011200:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828717.239562:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973011328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828717.239573:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828717.239576:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828717.239578:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828717.239580:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5167 00000100:00080000:2.0:1787828717.239584:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828717.239587:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828717.239592:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828717.239601:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973011456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828717.239610:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828717.239613:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828717.239614:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828717.239616:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5167 00000100:00080000:2.0:1787828717.239618:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828717.239620:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828717.239623:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828717.239627:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828717.239634:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973011584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828717.239641:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828717.239644:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828717.239646:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828717.239649:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5167 00000100:00080000:2.0:1787828717.239652:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828717.239654:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828717.239656:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828717.239665:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973011712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828717.239671:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828717.239673:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828717.239674:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828717.239678:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5167 00000100:00080000:2.0:1787828717.239680:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828717.239685:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff9c1a49d8d680 x1874668973011200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828717.239685:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828717.239692:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828717.239705:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973011840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828717.239712:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973011200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828717.239713:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828717.239721:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828717.239724:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828717.239733:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5167 00000100:00080000:2.0:1787828717.239738:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828717.239739:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828717.239742:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787828717.239747:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:2.0:1787828717.239749:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828717.239753:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973011200:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828717.239762:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973011968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828717.239789:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973011328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828717.239812:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828717.239830:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff9c1a49d8ed00 x1874668973011328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828717.239840:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973011328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828717.239846:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828717.239848:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828717.239851:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973011328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828717.239855:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973011456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828717.239870:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828717.239877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff9c1a49d8fc00 x1874668973011456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828717.239884:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973011456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828717.239890:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828717.239892:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828717.239893:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973011456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828717.239897:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973011584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828717.239908:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828717.239913:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff9c1a49d8e580 x1874668973011584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828717.239920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973011584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828717.239926:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828717.239927:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828717.239929:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973011584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828717.239933:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973011712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828717.239943:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828717.239948:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff9c1a49d8c3c0 x1874668973011712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828717.239967:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973011712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828717.239973:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828717.239974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828717.239976:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973011712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828717.239980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973011840:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828717.239992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828717.239998:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff9c1a49d8e1c0 x1874668973011840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828717.240005:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973011840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828717.240010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828717.240012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828717.240014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973011840:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828717.240017:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973011968:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828717.240033:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828717.240039:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828717/real 1787828717] req@ffff9c1a49d8f0c0 x1874668973011968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828717.240045:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973011968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828717.240055:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828717.240057:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828717.240059:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973011968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828717.751435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828717.751443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828718.007454:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828718.775474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828718.775474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828719.031482:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828719.799423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828719.799425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828720.055593:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828720.823456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828720.823456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828721.079488:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828721.847424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828721.847434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828722.103468:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828722.359481:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828722.359490:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828722.359493:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828722.359501:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5172 00000100:00080000:0.0:1787828722.359510:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828722.359516:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828722.359554:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973012096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828722.359585:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828722.359588:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828722.359590:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828722.359594:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5172 00000100:00080000:0.0:1787828722.359598:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828722.359602:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828722.359611:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828722.359624:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973012224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828722.359636:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828722.359639:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973012096:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828722.359640:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828722.359642:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828722.359655:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5172 00000100:00080000:0.0:1787828722.359658:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828722.359660:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828722.359663:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828722.359674:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973012352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828722.359681:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828722.359684:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828722.359685:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828722.359688:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5172 00000100:00080000:0.0:1787828722.359690:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828722.359692:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828722.359695:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828722.359704:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973012480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828722.359711:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828722.359714:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828722.359716:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828722.359718:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5172 00000100:00080000:0.0:1787828722.359721:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828722.359723:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787828722.359733:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787828722.359749:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828722.359765:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973012608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828722.359772:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828722.359775:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828722.359777:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828722.359781:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5172 00000100:00080000:0.0:1787828722.359785:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828722.359790:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828722.359795:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828722.359801:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff9c1a788621c0 x1874668973012096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828722.359809:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973012736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828722.359856:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828722.359859:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828722.359861:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828722.359869:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5172 00000100:00100000:3.0:1787828722.359869:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973012096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828722.359876:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828722.359888:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828722.359893:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828722.359901:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973012864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828722.359905:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828722.359917:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828722.359924:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973012096:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828722.359957:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973012224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828722.359977:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828722.359987:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff9c1a78860780 x1874668973012224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828722.359996:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973012224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828722.360002:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828722.360004:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828722.360007:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973012224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828722.360010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973012352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828722.360022:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828722.360028:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff9c1a78861e00 x1874668973012352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828722.360036:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973012352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828722.360046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828722.360048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828722.360050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973012352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828722.360054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973012480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828722.360064:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828722.360070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff9c1a78862580 x1874668973012480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828722.360077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973012480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828722.360083:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828722.360085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828722.360087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973012480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828722.360090:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973012608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828722.360100:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828722.360105:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff9c1a78861680 x1874668973012608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828722.360113:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973012608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828722.360118:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828722.360121:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828722.360123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973012608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828722.360128:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973012736:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828722.360139:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828722.360144:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff9c1a788630c0 x1874668973012736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828722.360151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973012736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828722.360156:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828722.360158:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828722.360160:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973012736:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828722.360164:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973012864:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828722.360173:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828722.360179:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828722/real 1787828722] req@ffff9c1a78863c00 x1874668973012864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828722.360187:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973012864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828722.360192:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828722.360194:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828722.360195:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973012864:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828722.871445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828722.871445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828723.127525:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828723.895430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828723.895445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828724.151528:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828724.919521:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828724.919521:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828725.175587:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828725.943435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828725.943435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828726.199529:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828726.967445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828726.967445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828727.223524:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828727.479471:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828727.479480:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828727.479483:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828727.479492:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5177 00000100:00080000:3.0:1787828727.479501:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828727.479523:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828727.479558:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973012992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828727.479600:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828727.479602:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828727.479604:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828727.479607:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5177 00000100:00080000:3.0:1787828727.479610:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828727.479614:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828727.479622:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828727.479651:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973012992:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828727.479651:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973013120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828727.479669:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828727.479672:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828727.479673:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828727.479676:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5177 00000100:00080000:3.0:1787828727.479679:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828727.479682:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828727.479686:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828727.479696:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973013248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828727.479727:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828727.479731:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828727.479733:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828727.479752:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5177 00000100:00080000:3.0:1787828727.479755:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828727.479758:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787828727.479763:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828727.479765:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828727.479777:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973013376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828727.479784:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828727.479787:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828727.479788:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828727.479791:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5177 00000100:00080000:3.0:1787828727.479793:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828727.479796:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828727.479800:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828727.479802:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828727/real 1787828727] req@ffff9c1a7895c780 x1874668973012992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828727.479810:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973013504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828727.479819:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828727.479823:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973012992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828727.479823:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828727.479824:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828727.479833:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5177 00000100:00080000:3.0:1787828727.479839:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828727.479840:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828727.479843:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787828727.479848:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787828727.479849:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828727.479853:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973012992:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828727.479859:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973013632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828727.479866:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828727.479868:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828727.479870:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828727.479873:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5177 00000100:00080000:3.0:1787828727.479875:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828727.479877:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973013120:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828727.479882:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828727.479886:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828727.479895:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973013760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828727.479904:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828727.479916:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828727/real 1787828727] req@ffff9c1a7895e940 x1874668973013120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828727.479926:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973013120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828727.479933:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828727.479936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828727.479938:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973013120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828727.479942:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973013248:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828727.479953:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828727.479958:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828727/real 1787828727] req@ffff9c1a7895c3c0 x1874668973013248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828727.479970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973013248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828727.479975:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828727.479977:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828727.479978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973013248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828727.479982:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973013376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828727.479992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828727.479997:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828727/real 1787828727] req@ffff9c1a7895cf00 x1874668973013376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828727.480004:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973013376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828727.480009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828727.480011:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828727.480013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973013376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828727.480016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973013504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828727.480026:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828727.480031:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828727/real 1787828727] req@ffff9c1a7895f480 x1874668973013504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828727.480038:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973013504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828727.480048:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828727.480050:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828727.480052:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973013504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828727.480055:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973013632:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828727.480067:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828727.480088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828727/real 1787828727] req@ffff9c1a7895d2c0 x1874668973013632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828727.480096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973013632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828727.480102:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828727.480104:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828727.480106:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973013632:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828727.480119:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973013760:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828727.480155:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828727.480163:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828727/real 1787828727] req@ffff9c1a7895c000 x1874668973013760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828727.480170:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973013760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828727.480176:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828727.480178:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828727.480181:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973013760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828727.991418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828727.991418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828728.247543:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828729.015435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828729.015435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828729.271561:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828730.039447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828730.039552:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828730.295472:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828731.063441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828731.063441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828731.319531:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828732.087452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828732.087453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828732.343539:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828732.599454:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828732.599462:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828732.599466:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828732.599474:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5182 00000100:00080000:2.0:1787828732.599484:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828732.599490:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828732.599527:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973013888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828732.599553:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828732.599557:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828732.599558:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828732.599562:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5182 00000100:00080000:2.0:1787828732.599565:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828732.599569:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828732.599576:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828732.599586:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973014016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828732.599594:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828732.599596:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828732.599598:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828732.599601:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5182 00000100:00080000:2.0:1787828732.599604:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828732.599607:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828732.599610:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828732.599617:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973014144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828732.599625:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828732.599627:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828732.599629:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828732.599633:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973013888:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828732.599634:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5182 00000100:00080000:2.0:1787828732.599637:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828732.599650:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828732.599655:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828732.599664:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973014272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828732.599682:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828732.599684:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828732.599686:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828732.599689:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5182 00000100:00080000:2.0:1787828732.599692:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828732.599695:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828732.599699:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828732.599709:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973014400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828732.599716:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828732.599718:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828732.599719:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828732.599721:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5182 00000100:00080000:2.0:1787828732.599723:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828732.599726:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787828732.599731:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787828732.599733:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828732.599744:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973014528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828732.599762:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828732.599765:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828732.599766:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828732.599769:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5182 00000100:00080000:2.0:1787828732.599772:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828732.599775:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828732.599778:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828732.599782:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828732/real 1787828732] req@ffff9c1a49d8de00 x1874668973013888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828732.599789:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973014656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828732.599806:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973013888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828732.599816:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828732.599819:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828732.599822:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973013888:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828732.599827:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973014016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828732.599843:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828732.599852:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828732/real 1787828732] req@ffff9c1a49d8c780 x1874668973014016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828732.599859:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973014016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828732.599865:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828732.599867:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828732.599869:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973014016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828732.599887:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973014144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828732.599898:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828732.599904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828732/real 1787828732] req@ffff9c1a49d8c000 x1874668973014144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828732.599912:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973014144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828732.599917:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828732.599919:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828732.599920:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973014144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828732.599924:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973014272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828732.599933:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828732.599939:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828732/real 1787828732] req@ffff9c1a49d8cf00 x1874668973014272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828732.599946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973014272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828732.599951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828732.599953:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828732.599955:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973014272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828732.599958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973014400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828732.599968:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828732.599973:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828732/real 1787828732] req@ffff9c1a49d8da40 x1874668973014400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828732.599980:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973014400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828732.599986:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828732.599988:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828732.599990:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973014400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828732.599993:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973014528:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828732.600003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828732.600008:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828732/real 1787828732] req@ffff9c1a49d8cb40 x1874668973014528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828732.600015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973014528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828732.600020:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828732.600021:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828732.600023:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973014528:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828732.600026:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973014656:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828732.600036:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828732.600041:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828732/real 1787828732] req@ffff9c1a49d8f0c0 x1874668973014656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828732.600048:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973014656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828732.600053:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828732.600054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828732.600057:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973014656:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828733.111472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828733.111472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828733.367610:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828734.135454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828734.135454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828734.391486:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828735.159414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828735.159462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828735.415744:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828736.183458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828736.183503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828736.439701:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828737.207437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828737.207437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828737.463462:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828737.719509:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828737.719516:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828737.719520:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828737.719528:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5188 00000100:00080000:3.0:1787828737.719538:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828737.719543:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828737.719574:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973014784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828737.719595:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828737.719598:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828737.719600:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828737.719603:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5188 00000100:00080000:3.0:1787828737.719605:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828737.719607:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828737.719614:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828737.719623:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973014912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828737.719641:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828737.719644:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828737.719645:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828737.719648:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5188 00000100:00080000:3.0:1787828737.719651:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828737.719653:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828737.719657:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828737.719665:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973015040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828737.719674:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828737.719676:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828737.719677:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828737.719680:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5188 00000100:00100000:0.0:1787828737.719685:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973014784:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828737.719686:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828737.719692:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828737.719696:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828737.719703:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973015168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828737.719710:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828737.719713:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828737.719714:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828737.719717:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5188 00000100:00080000:3.0:1787828737.719719:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828737.719721:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828737.719725:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828737.719731:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973015296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828737.719737:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828737.719739:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828737.719740:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828737.719743:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5188 00000100:00080000:3.0:1787828737.719746:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828737.719749:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828737.719753:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828737.719758:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828737.719766:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973015424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828737.719776:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828737.719779:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828737.719780:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828737.719783:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5188 00000100:00080000:3.0:1787828737.719786:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828737.719791:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828737/real 1787828737] req@ffff9c1a7895e580 x1874668973014784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828737.719792:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828737.719798:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828737.719810:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973015552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828737.719816:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973014784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828737.719836:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828737.719840:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828737.719843:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973014784:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828737.719848:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973014912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828737.719865:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828737.719874:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828737/real 1787828737] req@ffff9c1a7895c000 x1874668973014912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828737.719882:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973014912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828737.719887:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828737.719889:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828737.719892:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973014912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828737.719895:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973015040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828737.719907:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828737.719913:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828737/real 1787828737] req@ffff9c1a7895d2c0 x1874668973015040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828737.719928:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973015040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828737.719934:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828737.719936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828737.719938:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973015040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828737.719958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973015168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828737.719970:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828737.719977:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828737/real 1787828737] req@ffff9c1a7895f480 x1874668973015168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828737.719985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973015168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828737.719991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828737.719993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828737.719995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973015168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828737.719999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973015296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828737.720008:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828737.720013:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828737/real 1787828737] req@ffff9c1a7895cf00 x1874668973015296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828737.720025:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973015296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828737.720030:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828737.720032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828737.720034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973015296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828737.720037:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973015424:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828737.720047:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828737.720053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828737/real 1787828737] req@ffff9c1a7895c3c0 x1874668973015424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828737.720061:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973015424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828737.720066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828737.720067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828737.720069:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973015424:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828737.720072:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973015552:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828737.720081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828737.720086:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828737/real 1787828737] req@ffff9c1a7895e940 x1874668973015552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828737.720093:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973015552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828737.720103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828737.720105:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828737.720107:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973015552:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828738.231475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828738.231475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828738.487504:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828739.255475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828739.255502:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828739.511599:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828740.279432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828740.279437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828740.535687:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828741.303490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828741.303521:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828741.559820:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828742.327475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828742.327494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828742.583669:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828742.839514:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828742.839527:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828742.839534:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828742.839564:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5193 00000100:00080000:3.0:1787828742.839579:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828742.839588:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828742.839673:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973015680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828742.839714:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828742.839718:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828742.839721:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828742.839727:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5193 00000100:00080000:3.0:1787828742.839731:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828742.839738:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828742.839752:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828742.839790:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973015808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828742.839804:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828742.839810:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828742.839814:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828742.839838:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5193 00000100:00080000:3.0:1787828742.839846:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828742.839855:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828742.839864:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828742.839890:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973015936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828742.839915:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828742.839918:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828742.839920:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828742.839926:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5193 00000100:00080000:3.0:1787828742.839931:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828742.839937:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828742.839943:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828742.839957:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973016064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828742.839967:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973015680:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787828742.840122:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828742.840209:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828742/real 1787828742] req@ffff9c1a7895c780 x1874668973015680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828742.840247:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973015680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828742.840266:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828742.840286:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828742.840292:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973015680:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828742.840308:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973015808:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828742.840334:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828742.840339:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828742.840342:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828742.840401:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828742.840427:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828742/real 1787828742] req@ffff9c1a7895de00 x1874668973015808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828742.840458:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973015808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828742.840478:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5193 00000100:00080000:3.0:1787828742.840492:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828742.840501:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828742.840520:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828742.840521:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828742.840540:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828742.840542:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973016192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828742.840559:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973015808:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828742.840582:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828742.840588:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828742.840590:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828742.840619:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5193 00000100:00100000:1.0:1787828742.840622:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973015936:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828742.840626:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828742.840656:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828742.840664:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828742.840682:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973016320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828742.840692:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828742.840696:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828742.840699:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828742.840716:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828742.840716:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5193 00000100:00080000:3.0:1787828742.840731:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828742.840736:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828742.840743:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828742.840767:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973016448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828742.840769:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828742/real 1787828742] req@ffff9c1a7895e1c0 x1874668973015936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828742.840820:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973015936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828742.840839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828742.840845:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828742.840851:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973015936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828742.840905:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973016064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828742.840929:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828742.840940:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828742/real 1787828742] req@ffff9c1a7895fc00 x1874668973016064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828742.840952:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973016064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828742.840979:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828742.840983:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828742.840987:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973016064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828742.840995:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973016192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828742.841017:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828742.841029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828742/real 1787828742] req@ffff9c1a7895f840 x1874668973016192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828742.841043:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973016192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828742.841052:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828742.841055:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828742.841059:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973016192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828742.841066:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973016320:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828742.841084:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828742.841095:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828742/real 1787828742] req@ffff9c1a7895f0c0 x1874668973016320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828742.841108:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973016320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828742.841127:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828742.841131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828742.841134:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973016320:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828742.841141:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973016448:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828742.841160:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828742.841170:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828742/real 1787828742] req@ffff9c1a7895cb40 x1874668973016448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828742.841183:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973016448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828742.841194:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828742.841197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828742.841203:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973016448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828743.351500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828743.351622:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828743.607628:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828744.375537:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828744.375565:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828744.631509:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828745.399447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828745.399529:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828745.655676:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828746.423530:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828746.423537:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828746.679707:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828747.447467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828747.447467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828747.703530:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828747.959456:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828747.959466:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828747.959471:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828747.959481:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5198 00000100:00080000:2.0:1787828747.959492:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828747.959499:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828747.959543:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973016576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828747.959582:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828747.959585:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828747.959586:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828747.959590:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5198 00000100:00080000:2.0:1787828747.959593:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828747.959596:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828747.959607:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828747.959619:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973016704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828747.959627:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828747.959630:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828747.959631:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828747.959637:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973016576:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828747.959637:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5198 00000100:00080000:2.0:1787828747.959639:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828747.959645:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828747.959651:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828747.959656:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973016832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828747.959663:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828747.959666:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828747.959667:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828747.959669:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5198 00000100:00080000:2.0:1787828747.959672:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828747.959674:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828747.959677:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828747.959685:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973016960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828747.959691:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828747.959693:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828747.959695:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828747.959697:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5198 00000100:00080000:2.0:1787828747.959700:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828747.959703:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828747.959706:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828747.959712:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973017088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828747.959718:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828747.959719:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828747.959721:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828747.959724:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5198 00000800:00000100:1.0:1787828747.959730:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828747.959730:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828747.959739:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828747.959746:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828747.959761:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973017216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828747.959771:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828747.959775:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828747.959777:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828747.959782:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5198 00000100:00100000:1.0:1787828747.959787:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828747/real 1787828747] req@ffff9c1a49d8e1c0 x1874668973016576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828747.959788:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828747.959795:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828747.959801:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828747.959815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973017344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828747.959820:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973016576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828747.959836:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828747.959843:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828747.959847:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973016576:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828747.959872:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973016704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828747.959892:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828747.959902:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828747/real 1787828747] req@ffff9c1a49d8c3c0 x1874668973016704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828747.959910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973016704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828747.959916:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828747.959918:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828747.959921:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973016704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828747.959924:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973016832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828747.959935:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828747.959940:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828747/real 1787828747] req@ffff9c1a49d8e580 x1874668973016832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828747.959948:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973016832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828747.959953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828747.959955:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828747.959958:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973016832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828747.959961:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973016960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828747.959972:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828747.959978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828747/real 1787828747] req@ffff9c1a49d8fc00 x1874668973016960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828747.959986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973016960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828747.959992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828747.959993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828747.959995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973016960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828747.959998:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973017088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828747.960009:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828747.960014:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828747/real 1787828747] req@ffff9c1a49d8ed00 x1874668973017088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828747.960022:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973017088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828747.960027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828747.960029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828747.960031:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973017088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828747.960034:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973017216:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828747.960045:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828747.960050:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828747/real 1787828747] req@ffff9c1a49d8d680 x1874668973017216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828747.960063:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973017216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828747.960068:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828747.960069:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828747.960071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973017216:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828747.960074:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973017344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828747.960084:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828747.960089:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828747/real 1787828747] req@ffff9c1a49d8f0c0 x1874668973017344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828747.960096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973017344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828747.960102:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828747.960104:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828747.960105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973017344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828748.471420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828748.471434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828748.727549:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828749.495483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828749.495500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828749.751624:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828750.519424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828750.519534:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828750.775700:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828751.543610:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828751.543612:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828751.799622:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828752.567513:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828752.567514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828752.823582:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828753.079508:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828753.079522:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828753.079539:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828753.079569:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5203 00000100:00080000:2.0:1787828753.079582:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828753.079590:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828753.079660:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973017472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828753.079714:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828753.079718:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828753.079719:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828753.079726:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5203 00000100:00080000:2.0:1787828753.079732:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828753.079739:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828753.079770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973017472:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787828753.079770:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828753.079797:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973017600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828753.079834:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828753.079838:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828753.079840:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828753.079847:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5203 00000100:00080000:2.0:1787828753.079852:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828753.079857:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828753.079863:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828753.079871:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828753.079876:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973017728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828753.079888:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828753.079892:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828753.079894:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828753.079898:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5203 00000100:00080000:2.0:1787828753.079902:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828753.079929:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828753.079934:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828753.079937:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff9c1a49d8cb40 x1874668973017472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828753.079946:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973017856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828753.079999:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828753.080002:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828753.080005:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828753.080011:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5203 00000100:00080000:2.0:1787828753.080015:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828753.080020:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828753.080021:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973017472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828753.080026:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828753.080045:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828753.080053:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973017984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828753.080059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787828753.080080:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828753.080084:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828753.080087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973017472:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828753.080088:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828753.080108:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5203 00000100:00080000:2.0:1787828753.080113:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828753.080117:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828753.080124:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828753.080126:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973017600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828753.080170:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973018112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828753.080194:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828753.080220:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828753.080226:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff9c1a49d8da40 x1874668973017600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828753.080227:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828753.080232:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828753.080248:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5203 00000100:00080000:2.0:1787828753.080258:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828753.080263:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973017600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828753.080265:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828753.080292:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828753.080295:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828753.080297:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828753.080305:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973017600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828753.080312:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973017728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828753.080319:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973018240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828753.080339:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828753.080354:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff9c1a49d8cf00 x1874668973017728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.080366:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973017728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.080417:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.080422:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828753.080426:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973017728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828753.080433:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973017856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828753.080453:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828753.080462:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff9c1a49d8c000 x1874668973017856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.080470:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973017856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.080476:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.080479:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828753.080481:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973017856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828753.080485:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973017984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828753.080497:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828753.080503:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff9c1a49d8c780 x1874668973017984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.080510:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973017984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.080515:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.080517:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828753.080520:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973017984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828753.080554:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973018112:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828753.080571:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828753.080579:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff9c1a49d8de00 x1874668973018112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.080586:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973018112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.080592:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.080594:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828753.080596:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973018112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828753.080600:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973018240:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828753.080610:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828753.080616:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828753/real 1787828753] req@ffff9c1a49d8cb40 x1874668973018240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828753.080624:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973018240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828753.080629:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828753.080631:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828753.080633:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973018240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828753.591581:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828753.591650:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828753.847600:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828754.615489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828754.615489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828754.871586:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828755.639465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828755.639465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828755.895680:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828756.663437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828756.663457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828756.919583:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828757.687475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828757.687476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828757.943510:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828758.199465:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828758.199473:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828758.199476:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828758.199485:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5208 00000100:00080000:3.0:1787828758.199505:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828758.199512:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828758.199583:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973018368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828758.199619:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828758.199626:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828758.199629:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828758.199636:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5208 00000100:00080000:3.0:1787828758.199644:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828758.199651:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828758.199665:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828758.199688:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973018496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828758.199701:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973018368:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828758.199705:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828758.199710:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828758.199713:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828758.199719:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5208 00000100:00080000:3.0:1787828758.199725:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828758.199731:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828758.199738:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828758.199753:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973018624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828758.199794:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828758.199794:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828758.199798:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828758.199799:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828758.199802:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5208 00000100:00080000:3.0:1787828758.199805:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828758.199809:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828758.199813:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828758.199823:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973018752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828758.199833:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff9c1a7895da40 x1874668973018368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828758.199844:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828758.199847:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828758.199849:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828758.199852:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973018368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828758.199852:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5208 00000100:00080000:3.0:1787828758.199856:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828758.199870:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828758.199872:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828758.199874:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828758.199876:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828758.199880:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973018368:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828758.199883:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973018880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828758.199891:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828758.199893:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828758.199895:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828758.199898:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5208 00000100:00080000:3.0:1787828758.199901:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828758.199902:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973018496:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828758.199905:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828758.199910:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828758.199919:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973019008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828758.199923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828758.199940:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828758.199944:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828758.199945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff9c1a7895cb40 x1874668973018496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828758.199946:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828758.199949:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5208 00000100:00080000:3.0:1787828758.199953:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828758.199957:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828758.199958:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973018496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828758.199962:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828758.199967:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.199970:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828758.199971:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973019136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828758.199974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973018496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828758.199980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973018624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828758.199998:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828758.200016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff9c1a7895f0c0 x1874668973018624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828758.200037:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973018624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828758.200043:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.200045:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828758.200047:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973018624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828758.200052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973018752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828758.200067:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828758.200074:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff9c1a7895f840 x1874668973018752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828758.200082:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973018752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828758.200087:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.200089:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828758.200091:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973018752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828758.200095:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973018880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828758.200104:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828758.200109:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff9c1a7895fc00 x1874668973018880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828758.200117:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973018880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828758.200126:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.200128:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828758.200130:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973018880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828758.200140:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973019008:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828758.200150:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828758.200156:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff9c1a7895e1c0 x1874668973019008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828758.200163:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973019008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828758.200169:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.200171:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828758.200173:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973019008:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828758.200176:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973019136:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828758.200185:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828758.200190:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828758/real 1787828758] req@ffff9c1a7895de00 x1874668973019136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828758.200197:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973019136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828758.200201:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828758.200203:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828758.200206:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973019136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828758.711429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828758.711465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828758.967463:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828759.735425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828759.735426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828759.991553:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828760.759463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828760.759514:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828761.015541:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828761.783459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828761.783535:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828762.039522:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828762.807437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828762.807437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828763.063582:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828763.319538:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828763.319552:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828763.319558:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828763.319573:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5213 00000100:00080000:1.0:1787828763.319588:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828763.319598:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828763.319660:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973019264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828763.319702:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828763.319707:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828763.319709:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828763.319714:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5213 00000100:00080000:1.0:1787828763.319719:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828763.319731:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828763.319746:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828763.319768:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973019392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828763.319770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973019264:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828763.319805:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828763.319810:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828763.319812:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828763.319818:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5213 00000100:00080000:1.0:1787828763.319822:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828763.319827:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828763.319832:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828763.319843:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973019520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828763.319856:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828763.319861:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828763.319864:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828763.319870:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5213 00000100:00080000:1.0:1787828763.319876:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828763.319883:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828763.319890:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828763.319907:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973019648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828763.319923:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828763.319928:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828763.319932:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828763.319939:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5213 00000100:00080000:1.0:1787828763.319945:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828763.319952:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828763.319959:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828763.319976:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973019776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828763.319990:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828763.319996:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828763.319999:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828763.320007:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5213 00000100:00080000:1.0:1787828763.320013:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828763.320026:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787828763.320026:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:1.0:1787828763.320041:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828763.320064:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973019904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828763.320095:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff9c1a78a78000 x1874668973019264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828763.320101:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828763.320115:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828763.320122:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828763.320134:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5213 00000100:00080000:1.0:1787828763.320140:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828763.320144:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973019264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828763.320150:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828763.320162:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828763.320181:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828763.320187:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973020032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828763.320194:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828763.320200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973019264:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828763.320253:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973019392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828763.320293:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828763.320309:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff9c1a78a7a580 x1874668973019392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828763.320331:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973019392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828763.320407:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828763.320412:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828763.320416:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973019392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828763.320425:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973019520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828763.320475:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828763.320492:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff9c1a78a78f00 x1874668973019520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828763.320505:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973019520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828763.320514:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828763.320518:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828763.320521:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973019520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828763.320527:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973019648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828763.320557:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828763.320574:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff9c1a78a7b480 x1874668973019648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828763.320594:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973019648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828763.320609:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828763.320615:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828763.320621:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973019648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828763.320632:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973019776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828763.320658:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828763.320670:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff9c1a78a79a40 x1874668973019776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828763.320683:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973019776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828763.320692:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828763.320695:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828763.320716:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973019776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828763.320738:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973019904:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828763.320767:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828763.320780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff9c1a78a792c0 x1874668973019904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828763.320795:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973019904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828763.320811:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828763.320817:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828763.320824:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973019904:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828763.320836:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973020032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828763.320872:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828763.320895:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828763/real 1787828763] req@ffff9c1a78a7b840 x1874668973020032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828763.320920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973020032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828763.320931:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828763.320935:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828763.320939:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973020032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828763.831463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828763.831558:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828764.087675:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828764.855438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828764.855438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828765.111577:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828765.879469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828765.879521:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828766.135529:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828766.903446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828766.903555:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828767.159485:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828767.927449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828767.927495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828768.183554:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828768.439477:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828768.439485:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828768.439489:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828768.439498:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5218 00000100:00080000:1.0:1787828768.439508:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828768.439514:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828768.439548:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973020160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828768.439573:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828768.439576:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828768.439577:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828768.439580:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5218 00000100:00080000:1.0:1787828768.439582:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828768.439585:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828768.439592:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828768.439602:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973020288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828768.439612:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828768.439616:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828768.439618:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828768.439623:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5218 00000100:00080000:1.0:1787828768.439627:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828768.439632:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828768.439641:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828768.439652:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973020160:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828768.439662:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973020416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828768.439678:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828768.439682:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828768.439686:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828768.439691:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5218 00000100:00080000:1.0:1787828768.439695:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828768.439700:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828768.439707:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828768.439722:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973020544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828768.439732:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828768.439736:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828768.439739:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828768.439743:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5218 00000100:00080000:1.0:1787828768.439747:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828768.439751:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828768.439757:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828768.439772:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973020672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828768.439784:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828768.439787:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973020288:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828768.439795:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828768.439799:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828768.439805:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5218 00000100:00080000:1.0:1787828768.439813:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828768.439820:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828768.439828:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828768.439835:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973020416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828768.439846:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973020800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828768.439863:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828768.439865:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973020544:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828768.439875:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828768.439879:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828768.439886:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5218 00000100:00100000:3.0:1787828768.439891:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973020672:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828768.439895:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828768.439904:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828768.439912:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828768.439913:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973020800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828768.439942:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973020928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828768.439971:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973020928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828768.951404:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828768.951447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828769.207498:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828769.975504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828769.975504:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828770.231513:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828770.999460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828770.999460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828771.255553:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828771.511693:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787828771.511773:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828771.511809:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828771.511879:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828771.511895:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828771.511912:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828771.511925:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828771.511944:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828771.511962:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787828771.512098:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828771] req@ffff9c1a78a7bc00 x1874668973020928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.512131:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973020928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.512145:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.512151:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828771.512156:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973020928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828771.512166:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828771] req@ffff9c1a78a78780 x1874668973020800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.512178:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973020800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.512186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.512190:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828771.512193:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973020800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828771.512200:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828771] req@ffff9c1a78a7a940 x1874668973020416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.512213:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973020416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.512222:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.512225:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828771.512229:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973020416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828771.512236:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828771] req@ffff9c1a78a783c0 x1874668973020544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.512247:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973020544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.512257:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.512261:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828771.512264:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973020544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828771.512272:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828771] req@ffff9c1a78a7a1c0 x1874668973020672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.512285:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973020672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.512294:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.512297:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828771.512301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973020672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828771.512307:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828771] req@ffff9c1a78a7b0c0 x1874668973020160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.512320:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973020160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.512327:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.512331:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828771.512334:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973020160:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828771.512339:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828768/real 1787828771] req@ffff9c1a78a79e00 x1874668973020288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828771.512972:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973020288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828771.512983:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828771.512988:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828771.512992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973020288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828772.023493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828772.023558:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828772.279562:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828773.047493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828773.047493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828773.303537:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828773.559433:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828773.559444:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828773.559448:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828773.559459:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5223 00000100:00080000:2.0:1787828773.559471:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828773.559479:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828773.559520:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973021056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828773.559553:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828773.559558:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828773.559560:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828773.559573:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5223 00000100:00080000:2.0:1787828773.559577:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828773.559581:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828773.559592:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828773.559603:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973021056:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828773.559607:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973021184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828773.559617:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828773.559619:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828773.559621:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828773.559623:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5223 00000100:00080000:2.0:1787828773.559626:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828773.559628:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828773.559632:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828773.559640:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973021312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828773.559648:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828773.559651:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828773.559652:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828773.559655:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5223 00000100:00080000:2.0:1787828773.559667:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828773.559671:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828773.559674:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828773.559679:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828773.559683:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973021440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828773.559691:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828773.559694:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828773.559695:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828773.559699:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5223 00000100:00080000:2.0:1787828773.559701:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828773.559703:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828773.559706:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828773.559714:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973021568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828773.559721:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828773.559725:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787828773.559725:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828773/real 1787828773] req@ffff9c1a49d8f0c0 x1874668973021056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828773.559730:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828773.559738:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5223 00000100:00080000:2.0:1787828773.559744:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828773.559751:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828773.559757:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828773.559775:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973021696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828773.559777:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973021056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828773.559794:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828773.559803:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787828773.559805:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828773.559812:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973021056:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828773.559813:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828773.559817:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828773.559826:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5223 00000100:00080000:2.0:1787828773.559831:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828773.559836:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828773.559841:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828773.559852:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973021184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828773.559857:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973021824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828773.559887:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828773.559905:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828773/real 1787828773] req@ffff9c1a49d8d680 x1874668973021184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828773.559918:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973021184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828773.559926:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828773.559929:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828773.559933:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973021184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828773.559939:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973021312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828773.559959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828773.559970:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828773/real 1787828773] req@ffff9c1a49d8ed00 x1874668973021312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828773.559982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973021312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828773.559995:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828773.559998:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828773.560012:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973021312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828773.560018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973021440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828773.560037:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828773.560047:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828773/real 1787828773] req@ffff9c1a49d8fc00 x1874668973021440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828773.560058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973021440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828773.560067:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828773.560082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828773.560085:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973021440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828773.560091:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973021568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828773.560111:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828773.560123:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828773/real 1787828773] req@ffff9c1a49d8e580 x1874668973021568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828773.560136:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973021568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828773.560144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828773.560148:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828773.560152:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973021568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828773.560158:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973021696:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828773.560180:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828773.560193:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828773/real 1787828773] req@ffff9c1a49d8c3c0 x1874668973021696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828773.560206:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973021696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828773.560215:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828773.560218:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828773.560222:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973021696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828773.560254:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973021824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828773.560273:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828773.560284:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828773/real 1787828773] req@ffff9c1a49d8e1c0 x1874668973021824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828773.560297:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973021824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828773.560472:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828773.560477:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828773.560482:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973021824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828774.071454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828774.071454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828774.327575:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828775.095490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828775.095491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828775.351588:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828776.119487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828776.119569:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828776.375594:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828777.143471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828777.143471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828777.399549:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828778.167454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828778.167506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828778.423480:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828778.679499:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828778.679508:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828778.679512:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828778.679522:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5229 00000100:00080000:1.0:1787828778.679533:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828778.679549:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828778.679600:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973021952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828778.679636:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828778.679641:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828778.679644:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828778.679649:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5229 00000100:00080000:1.0:1787828778.679656:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828778.679662:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828778.679673:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828778.679688:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973021952:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828778.679697:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973022080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828778.679736:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828778.679740:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828778.679742:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828778.679748:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5229 00000100:00080000:1.0:1787828778.679753:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828778.679758:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828778.679764:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828778.679776:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973022208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828778.679787:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828778.679790:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828778.679792:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828778.679795:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5229 00000100:00080000:1.0:1787828778.679799:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828778.679803:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828778.679808:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828778.679814:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828778.679823:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973022336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828778.679834:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828778.679837:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828778.679838:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828778.679842:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5229 00000100:00080000:1.0:1787828778.679846:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828778.679849:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828778.679854:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828778.679867:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973022464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828778.679867:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828778/real 1787828778] req@ffff9c1a78a79e00 x1874668973021952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828778.679893:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828778.679901:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828778.679902:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828778.679905:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5229 00000100:00080000:1.0:1787828778.679909:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828778.679912:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828778.679916:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828778.679917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973021952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828778.679928:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973022592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828778.679932:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828778.679941:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828778.679949:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973021952:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828778.679963:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828778.679969:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828778.679972:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828778.679992:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973022080:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828778.679999:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5229 00000100:00080000:1.0:1787828778.680007:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828778.680013:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828778.680019:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828778.680026:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828778.680037:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973022720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828778.680045:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828778/real 1787828778] req@ffff9c1a78a7b0c0 x1874668973022080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828778.680069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973022080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828778.680083:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828778.680087:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828778.680091:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973022080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828778.680099:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973022208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828778.680138:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828778.680155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828778/real 1787828778] req@ffff9c1a78a7a1c0 x1874668973022208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828778.680166:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973022208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828778.680173:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828778.680175:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828778.680178:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973022208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828778.680181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973022336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828778.680194:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828778.680201:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828778/real 1787828778] req@ffff9c1a78a783c0 x1874668973022336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828778.680209:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973022336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828778.680215:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828778.680217:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828778.680219:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973022336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828778.680223:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973022464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828778.680233:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828778.680239:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828778/real 1787828778] req@ffff9c1a78a7a940 x1874668973022464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828778.680247:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973022464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828778.680252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828778.680254:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828778.680256:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973022464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828778.680260:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973022592:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828778.680271:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828778.680277:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828778/real 1787828778] req@ffff9c1a78a78780 x1874668973022592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828778.680290:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973022592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828778.680296:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828778.680298:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828778.680300:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973022592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828778.680313:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973022720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828778.680324:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828778.680330:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828778/real 1787828778] req@ffff9c1a78a7bc00 x1874668973022720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828778.680338:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973022720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828778.680344:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828778.680390:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828778.680394:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973022720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828779.191457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828779.191457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828779.447562:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828780.215439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828780.215454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828780.471447:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828781.239454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828781.239472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828781.495509:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828782.263438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828782.263480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828782.519458:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828783.287497:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828783.287503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828783.543485:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828783.799450:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828783.799458:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828783.799462:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828783.799470:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5234 00000100:00080000:3.0:1787828783.799479:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828783.799485:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828783.799517:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973022848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828783.799540:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828783.799543:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828783.799544:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828783.799548:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5234 00000100:00080000:3.0:1787828783.799550:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828783.799552:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828783.799560:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828783.799568:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973022976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828783.799575:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828783.799577:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828783.799579:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828783.799581:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5234 00000100:00080000:3.0:1787828783.799584:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828783.799586:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828783.799590:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828783.799597:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973023104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828783.799605:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828783.799607:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828783.799609:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828783.799614:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973022848:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828783.799614:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5234 00000100:00080000:3.0:1787828783.799617:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828783.799623:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828783.799628:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828783.799634:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973023232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828783.799639:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828783.799642:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828783.799643:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828783.799645:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5234 00000100:00080000:3.0:1787828783.799664:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828783.799667:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828783.799671:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828783.799680:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973023360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828783.799687:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828783.799690:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828783.799693:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828783.799696:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5234 00000100:00080000:3.0:1787828783.799699:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828783.799706:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828783.799706:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828783.799718:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828783.799734:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973023488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828783.799747:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828783.799751:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828783/real 1787828783] req@ffff9c1a7895c780 x1874668973022848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828783.799754:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828783.799759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828783.799773:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5234 00000100:00080000:3.0:1787828783.799780:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828783.799785:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828783.799791:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973022848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828783.799793:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828783.799810:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973023616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828783.799814:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828783.799827:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828783.799834:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973022848:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828783.799846:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973022976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828783.799871:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828783.799883:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828783/real 1787828783] req@ffff9c1a7895e940 x1874668973022976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828783.799893:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973022976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828783.799900:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828783.799903:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828783.799906:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973022976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828783.799944:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973023104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828783.799956:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828783.799963:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828783/real 1787828783] req@ffff9c1a7895c3c0 x1874668973023104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828783.799970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973023104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828783.799976:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828783.799992:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828783.799995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973023104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828783.800001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973023232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828783.800014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828783.800022:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828783/real 1787828783] req@ffff9c1a7895cf00 x1874668973023232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828783.800032:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973023232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828783.800039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828783.800041:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828783.800043:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973023232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828783.800048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973023360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828783.800060:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828783.800066:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828783/real 1787828783] req@ffff9c1a7895f480 x1874668973023360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828783.800075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973023360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828783.800082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828783.800084:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828783.800086:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973023360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828783.800091:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973023488:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828783.800103:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828783.800110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828783/real 1787828783] req@ffff9c1a7895d2c0 x1874668973023488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828783.800120:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973023488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828783.800127:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828783.800129:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828783.800131:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973023488:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828783.800151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973023616:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828783.800163:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828783.800168:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828783/real 1787828783] req@ffff9c1a7895c000 x1874668973023616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828783.800175:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973023616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828783.800181:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828783.800182:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828783.800184:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973023616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828784.311426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828784.311426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828784.567480:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828785.335451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828785.335451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828785.591543:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828786.359442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828786.359477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828786.615512:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828787.383446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828787.383489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828787.639514:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828788.407449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828788.407498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828788.663495:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828788.919418:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828788.919426:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828788.919429:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828788.919438:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5239 00000100:00080000:3.0:1787828788.919447:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828788.919452:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828788.919483:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973023744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828788.919510:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828788.919513:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828788.919514:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828788.919517:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5239 00000100:00080000:3.0:1787828788.919519:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828788.919522:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828788.919529:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828788.919540:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973023872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828788.919548:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828788.919550:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828788.919551:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828788.919554:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5239 00000100:00080000:3.0:1787828788.919556:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828788.919558:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828788.919562:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828788.919566:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973023744:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828788.919574:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973024000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828788.919595:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828788.919600:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828788.919602:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828788.919606:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5239 00000100:00080000:3.0:1787828788.919620:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828788.919625:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828788.919631:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787828788.919633:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828788.919647:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973024128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828788.919655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828788.919657:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828788.919658:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828788.919660:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5239 00000100:00080000:3.0:1787828788.919663:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828788.919665:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828788.919668:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828788.919681:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973024256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828788.919690:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828788/real 1787828788] req@ffff9c1a7895e580 x1874668973023744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828788.919691:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828788.919698:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828788.919701:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828788.919709:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5239 00000100:00080000:3.0:1787828788.919714:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828788.919717:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828788.919724:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828788.919725:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973023744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828788.919734:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973024384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828788.919744:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828788.919754:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828788.919756:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828788.919761:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828788.919765:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973023744:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828788.919766:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828788.919775:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5239 00000100:00080000:3.0:1787828788.919779:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828788.919781:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973023872:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828788.919785:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828788.919791:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828788.919814:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973024512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828788.919820:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828788.919839:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828788/real 1787828788] req@ffff9c1a7895c000 x1874668973023872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828788.919856:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973023872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828788.919866:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828788.919870:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828788.919873:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973023872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828788.919898:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973024000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828788.919915:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828788.919924:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828788/real 1787828788] req@ffff9c1a7895d2c0 x1874668973024000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828788.919936:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973024000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828788.919945:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828788.919947:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828788.919951:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973024000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828788.919957:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973024128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828788.919975:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828788.919986:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828788/real 1787828788] req@ffff9c1a7895f480 x1874668973024128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828788.920000:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973024128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828788.920009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828788.920012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828788.920016:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973024128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828788.920022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973024256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828788.920037:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828788.920046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828788/real 1787828788] req@ffff9c1a7895cf00 x1874668973024256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828788.920058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973024256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828788.920067:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828788.920071:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828788.920074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973024256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828788.920079:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973024384:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828788.920094:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828788.920103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828788/real 1787828788] req@ffff9c1a7895c3c0 x1874668973024384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828788.920116:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973024384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828788.920125:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828788.920127:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828788.920130:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973024384:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828788.920136:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973024512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828788.920151:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828788.920160:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828788/real 1787828788] req@ffff9c1a7895e940 x1874668973024512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828788.920172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973024512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828788.920181:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828788.920184:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828788.920188:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973024512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828789.431451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828789.431451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828789.687570:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828790.455472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828790.455508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828790.711528:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828791.479435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828791.479436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828791.735555:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828792.503515:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828792.503517:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828792.759477:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828793.527438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828793.527487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828793.783545:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828794.039466:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.039474:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828794.039478:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.039493:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5244 00000100:00080000:0.0:1787828794.039505:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828794.039511:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828794.039542:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973024640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828794.039569:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.039572:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828794.039573:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.039577:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5244 00000100:00080000:0.0:1787828794.039579:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828794.039581:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828794.039589:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828794.039599:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973024768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828794.039606:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.039609:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828794.039611:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.039613:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5244 00000100:00080000:0.0:1787828794.039615:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828794.039618:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828794.039620:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828794.039627:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973024896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828794.039634:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973024640:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828794.039637:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.039644:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828794.039646:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.039649:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5244 00000100:00080000:0.0:1787828794.039653:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828794.039655:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828794.039659:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828794.039667:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973025024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828794.039672:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.039675:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828794.039676:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.039679:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5244 00000100:00080000:0.0:1787828794.039682:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828794.039685:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828794.039687:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828794.039694:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973025152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828794.039700:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.039703:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828794.039704:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.039717:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5244 00000100:00080000:0.0:1787828794.039719:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828794.039724:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828794.039725:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828794.039731:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828794.039739:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973025280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828794.039750:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828794.039753:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828794.039754:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828794.039756:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5244 00000100:00080000:0.0:1787828794.039759:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828794.039761:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828794.039764:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828794.039773:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973025408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828794.039773:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff9c1a78860000 x1874668973024640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828794.039813:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973024640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828794.039841:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828794.039846:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828794.039851:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973024640:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828794.039859:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973024768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828794.039884:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828794.039899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff9c1a78863c00 x1874668973024768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828794.039912:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973024768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828794.039922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828794.039926:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828794.039930:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973024768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828794.039956:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973024896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828794.039977:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828794.039988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff9c1a788630c0 x1874668973024896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828794.040001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973024896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828794.040010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828794.040013:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828794.040017:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973024896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828794.040025:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973025024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828794.040056:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828794.040068:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff9c1a78861680 x1874668973025024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828794.040082:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973025024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828794.040091:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828794.040095:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828794.040099:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973025024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828794.040105:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973025152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828794.040126:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828794.040137:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff9c1a78862580 x1874668973025152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828794.040150:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973025152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828794.040159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828794.040163:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828794.040166:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973025152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828794.040172:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973025280:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828794.040192:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828794.040203:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff9c1a78861e00 x1874668973025280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828794.040216:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973025280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828794.040225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828794.040229:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828794.040232:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973025280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828794.040237:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973025408:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828794.040270:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828794.040291:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828794/real 1787828794] req@ffff9c1a78860780 x1874668973025408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828794.040304:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973025408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828794.040313:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828794.040316:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828794.040320:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973025408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828794.551464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828794.551481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828794.807500:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828795.575412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828795.575434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828795.831477:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828796.599475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828796.599484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828796.855542:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828797.623424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828797.623424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828797.879468:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828798.647461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828798.647493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828798.903549:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828799.159501:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828799.159511:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828799.159515:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.159526:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5249 00000100:00080000:0.0:1787828799.159536:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.159544:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828799.159581:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973025536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828799.159604:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828799.159607:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828799.159609:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.159613:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5249 00000100:00080000:0.0:1787828799.159616:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.159619:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828799.159628:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828799.159639:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973025664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828799.159647:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973025536:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828799.159668:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828799.159675:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828799.159676:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.159681:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5249 00000100:00080000:0.0:1787828799.159683:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.159686:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828799.159690:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828799.159702:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973025792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828799.159709:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828799.159713:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828799.159714:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.159718:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5249 00000100:00080000:0.0:1787828799.159721:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.159723:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828799.159727:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828799.159737:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973025920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828799.159744:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828799.159749:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828799.159750:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828799.159756:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.159760:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5249 00000100:00080000:0.0:1787828799.159763:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.159765:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828799.159768:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828799.159794:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973026048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828799.159800:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828799.159802:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828799.159804:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.159807:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5249 00000100:00080000:0.0:1787828799.159810:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.159814:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828799.159818:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff9c1a788621c0 x1874668973025536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828799.159819:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828799.159831:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973026176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828799.159842:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828799.159844:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828799.159846:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828799.159851:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5249 00000100:00100000:2.0:1787828799.159853:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973025536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828799.159854:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828799.159860:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828799.159864:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828799.159875:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973026304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828799.159878:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828799.159887:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828799.159893:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973025536:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828799.159914:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973025664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828799.159935:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828799.159945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff9c1a78862940 x1874668973025664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828799.159955:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973025664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828799.159961:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828799.159963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828799.159966:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973025664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828799.159970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973025792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828799.159982:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828799.159988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff9c1a78861a40 x1874668973025792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828799.160006:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973025792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828799.160012:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828799.160015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828799.160017:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973025792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828799.160021:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973025920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828799.160034:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828799.160040:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff9c1a78860f00 x1874668973025920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828799.160049:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973025920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828799.160055:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828799.160057:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828799.160076:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973025920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828799.160081:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973026048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828799.160094:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828799.160100:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff9c1a78862d00 x1874668973026048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828799.160109:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973026048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828799.160115:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828799.160117:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828799.160120:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973026048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828799.160124:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973026176:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828799.160136:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828799.160143:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff9c1a78860b40 x1874668973026176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828799.160152:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973026176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828799.160158:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828799.160160:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828799.160162:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973026176:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828799.160166:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973026304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828799.160187:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828799.160199:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828799/real 1787828799] req@ffff9c1a78863840 x1874668973026304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828799.160233:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973026304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828799.160244:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828799.160248:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828799.160251:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973026304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828799.671504:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828799.671504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828799.927513:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828800.695437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828800.695437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828800.951531:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828801.719470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828801.719470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828801.975494:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828802.743449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828802.743449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828802.999529:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828803.767410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828803.767417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828804.023514:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828804.279488:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828804.279500:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828804.279505:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828804.279517:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5254 00000100:00080000:3.0:1787828804.279546:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828804.279561:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828804.279598:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973026432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828804.279625:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828804.279628:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828804.279630:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828804.279634:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5254 00000100:00080000:3.0:1787828804.279637:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828804.279641:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828804.279651:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828804.279662:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973026560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828804.279668:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973026432:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828804.279674:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828804.279678:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828804.279679:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828804.279682:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5254 00000100:00080000:3.0:1787828804.279684:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828804.279687:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828804.279691:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828804.279700:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973026688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828804.279708:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828804.279711:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828804.279712:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828804.279715:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5254 00000100:00080000:3.0:1787828804.279728:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828804.279731:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828804.279734:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828804.279742:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828804.279745:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973026816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828804.279754:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828804.279757:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828804.279758:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828804.279762:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5254 00000100:00080000:3.0:1787828804.279767:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828804.279769:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828804.279773:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828804.279783:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973026944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828804.279791:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff9c1a7895c780 x1874668973026432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828804.279792:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828804.279814:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828804.279815:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828804.279820:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5254 00000100:00080000:3.0:1787828804.279823:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828804.279830:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828804.279841:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828804.279856:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973027072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828804.279877:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973026432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828804.279892:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828804.279897:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828804.279899:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828804.279900:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828804.279908:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5254 00000100:00080000:3.0:1787828804.279914:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828804.279915:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828804.279921:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828804.279926:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973026432:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787828804.279928:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828804.279944:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973027200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828804.279960:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973026560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828804.279991:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828804.280002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff9c1a7895de00 x1874668973026560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828804.280012:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973026560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828804.280019:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828804.280022:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828804.280025:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973026560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828804.280030:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973026688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828804.280042:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828804.280049:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff9c1a7895e1c0 x1874668973026688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828804.280058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973026688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828804.280064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828804.280066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828804.280068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973026688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828804.280073:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973026816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828804.280085:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828804.280092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff9c1a7895fc00 x1874668973026816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828804.280101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973026816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828804.280107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828804.280109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828804.280112:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973026816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828804.280116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973026944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828804.280128:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828804.280134:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff9c1a7895f840 x1874668973026944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828804.280143:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973026944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828804.280149:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828804.280151:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828804.280153:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973026944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828804.280157:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973027072:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828804.280202:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828804.280213:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff9c1a7895f0c0 x1874668973027072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828804.280223:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973027072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828804.280229:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828804.280232:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828804.280234:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973027072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828804.280238:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973027200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828804.280271:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828804.280280:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828804/real 1787828804] req@ffff9c1a7895cb40 x1874668973027200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828804.280289:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973027200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828804.280297:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828804.280300:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828804.280302:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973027200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828804.791440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828804.791440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828805.047534:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828805.815444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828805.815473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828806.071568:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828806.839468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828806.839469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828807.095685:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828807.863451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828807.863452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828808.119475:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828808.887453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828808.887508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828809.143488:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828809.399457:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.399466:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828809.399470:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.399479:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5259 00000100:00080000:0.0:1787828809.399489:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828809.399496:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828809.399530:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973027328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828809.399554:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.399558:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828809.399559:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.399562:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5259 00000100:00080000:0.0:1787828809.399565:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828809.399573:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828809.399581:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828809.399593:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973027456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828809.399602:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.399605:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828809.399607:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.399610:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5259 00000100:00080000:0.0:1787828809.399612:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828809.399615:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828809.399621:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828809.399623:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973027328:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828809.399633:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973027584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828809.399648:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.399651:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828809.399653:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.399656:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5259 00000100:00080000:0.0:1787828809.399661:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828809.399666:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828809.399672:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828809.399687:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973027712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828809.399699:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828809.399708:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.399714:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828809.399717:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.399723:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5259 00000100:00080000:0.0:1787828809.399727:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828809.399731:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828809.399736:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828809.399750:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973027840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828809.399750:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff9c1a788612c0 x1874668973027328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828809.399788:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.399794:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828809.399796:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828809.399800:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973027328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828809.399805:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5259 00000100:00080000:0.0:1787828809.399815:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828809.399822:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828809.399824:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828809.399828:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828809.399833:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828809.399840:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973027328:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828809.399847:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973027968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828809.399854:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973027456:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828809.399869:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828809.399874:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828809.399877:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828809.399885:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5259 00000100:00080000:0.0:1787828809.399891:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828809.399893:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828809.399902:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828809.399908:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828809.399915:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff9c1a78863840 x1874668973027456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828809.399923:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973028096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828809.399935:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973027456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828809.399944:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828809.399946:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828809.399950:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973027456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828809.399973:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973027584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828809.399990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828809.399999:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff9c1a78860b40 x1874668973027584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828809.400007:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973027584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828809.400014:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828809.400016:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828809.400018:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973027584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828809.400023:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973027712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828809.400035:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828809.400041:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff9c1a78862d00 x1874668973027712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828809.400050:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973027712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828809.400068:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828809.400069:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828809.400071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973027712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828809.400075:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973027840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828809.400086:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828809.400093:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff9c1a78860f00 x1874668973027840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828809.400101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973027840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828809.400107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828809.400109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828809.400111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973027840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828809.400116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973027968:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828809.400127:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828809.400164:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff9c1a78861a40 x1874668973027968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828809.400175:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973027968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828809.400182:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828809.400184:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828809.400186:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973027968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828809.400191:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973028096:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828809.400216:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828809.400222:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828809/real 1787828809] req@ffff9c1a78862940 x1874668973028096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828809.400231:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973028096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828809.400237:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828809.400239:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828809.400241:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973028096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828809.911490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828809.911497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828810.167606:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828810.935470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828810.935547:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828811.191563:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828811.959550:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828811.959565:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828812.215610:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828812.983497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828812.983526:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828813.239596:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828814.007543:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828814.007543:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828814.263611:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828814.519458:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828814.519468:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828814.519472:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828814.519483:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5264 00000100:00080000:0.0:1787828814.519502:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828814.519510:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828814.519547:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973028224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828814.519578:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828814.519583:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828814.519586:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828814.519592:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5264 00000100:00080000:0.0:1787828814.519598:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828814.519604:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828814.519617:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828814.519637:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973028352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828814.519654:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828814.519659:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828814.519662:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828814.519671:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973028224:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828814.519672:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5264 00000100:00080000:0.0:1787828814.519681:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828814.519690:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828814.519699:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828814.519716:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973028480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828814.519747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828814.519751:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828814.519754:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828814.519758:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5264 00000100:00080000:0.0:1787828814.519763:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828814.519769:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828814.519775:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828814.519805:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828814.519808:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973028608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828814.519819:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828814.519823:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828814.519824:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828814.519829:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5264 00000100:00080000:0.0:1787828814.519834:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828814.519838:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828814.519843:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828814.519871:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973028736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828814.519880:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828814.519883:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828814.519885:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828814.519892:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828814/real 1787828814] req@ffff9c1a788621c0 x1874668973028224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828814.519893:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5264 00000100:00080000:0.0:1787828814.519900:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828814.519908:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828814.519918:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828814.519963:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973028864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828814.519976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828814.519981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973028224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828814.519982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828814.520004:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828814.520022:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5264 00000100:00080000:0.0:1787828814.520029:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828814.520037:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828814.520043:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828814.520045:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828814.520053:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973028992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828814.520057:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828814.520072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973028224:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828814.520095:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973028352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828814.520167:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828814.520189:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828814/real 1787828814] req@ffff9c1a78860780 x1874668973028352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828814.520211:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973028352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828814.520225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828814.520231:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828814.520239:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973028352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828814.520275:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973028480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828814.520311:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828814.520335:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828814/real 1787828814] req@ffff9c1a78861e00 x1874668973028480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828814.520414:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973028480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828814.520440:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828814.520449:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828814.520455:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973028480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828814.520468:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973028608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828814.520501:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828814.520520:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828814/real 1787828814] req@ffff9c1a78862580 x1874668973028608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828814.520541:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973028608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828814.520556:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828814.520563:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828814.520571:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973028608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828814.520584:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973028736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828814.520623:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828814.520645:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828814/real 1787828814] req@ffff9c1a78861680 x1874668973028736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828814.520666:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973028736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828814.520679:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828814.520684:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828814.520690:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973028736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828814.520700:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973028864:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828814.520744:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828814.520762:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828814/real 1787828814] req@ffff9c1a788630c0 x1874668973028864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828814.520784:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973028864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828814.520797:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828814.520803:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828814.520808:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973028864:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828814.520819:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973028992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828814.520860:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828814.520885:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828814/real 1787828814] req@ffff9c1a78863c00 x1874668973028992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828814.520908:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973028992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828814.520924:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828814.520947:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828814.520955:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973028992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828815.031463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828815.031467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828815.287507:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828816.055430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828816.055448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828816.311503:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828817.079430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828817.079430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828817.335564:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828818.103465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828818.103471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828818.359572:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828819.127424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828819.127457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828819.383607:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828819.639489:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828819.639499:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828819.639504:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828819.639514:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5269 00000100:00080000:0.0:1787828819.639525:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828819.639532:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828819.639572:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973029120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828819.639603:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828819.639607:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828819.639609:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828819.639613:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5269 00000100:00080000:0.0:1787828819.639616:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828819.639619:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828819.639629:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828819.639645:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973029248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828819.639662:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828819.639668:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828819.639670:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828819.639674:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973029120:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828819.639678:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5269 00000100:00080000:0.0:1787828819.639689:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828819.639695:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828819.639704:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828819.639724:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973029376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828819.639737:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828819.639741:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828819.639743:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828819.639747:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5269 00000100:00080000:0.0:1787828819.639751:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828819.639752:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828819.639761:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828819.639768:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828819.639791:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973029504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828819.639801:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828819.639805:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828819.639807:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828819.639811:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5269 00000100:00080000:0.0:1787828819.639814:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828819.639821:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828819.639821:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828819/real 1787828819] req@ffff9c1a78860000 x1874668973029120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828819.639829:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828819.639844:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973029632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828819.639854:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828819.639858:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828819.639864:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828819.639865:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973029120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828819.639870:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5269 00000100:00080000:0.0:1787828819.639877:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828819.639884:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828819.639892:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828819.639901:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828819.639907:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973029760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828819.639915:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828819.639923:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828819.639930:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828819.639931:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973029120:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828819.639932:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828819.639943:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5269 00000100:00080000:0.0:1787828819.639947:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828819.639950:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828819.639954:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828819.639964:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973029888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828819.639980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973029248:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828819.640033:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828819.640054:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828819/real 1787828819] req@ffff9c1a78863c00 x1874668973029248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.640092:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973029248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.640106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.640111:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828819.640116:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973029248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828819.640125:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973029376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828819.640157:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828819.640175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828819/real 1787828819] req@ffff9c1a788630c0 x1874668973029376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.640192:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973029376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.640205:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.640209:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828819.640214:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973029376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828819.640223:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973029504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828819.640252:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828819.640269:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828819/real 1787828819] req@ffff9c1a78861680 x1874668973029504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.640286:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973029504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.640300:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.640305:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828819.640309:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973029504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828819.640318:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973029632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828819.640386:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828819.640405:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828819/real 1787828819] req@ffff9c1a78862580 x1874668973029632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.640431:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973029632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.640444:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.640449:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828819.640454:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973029632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828819.640462:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973029760:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828819.640490:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828819.640518:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828819/real 1787828819] req@ffff9c1a78861e00 x1874668973029760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.640538:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973029760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.640551:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.640556:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828819.640561:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973029760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828819.640592:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973029888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828819.640630:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828819.640646:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828819/real 1787828819] req@ffff9c1a78860780 x1874668973029888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828819.640665:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973029888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828819.640687:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828819.640692:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828819.640697:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973029888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828820.151522:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828820.151528:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828820.407574:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828821.175452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828821.175452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828821.431540:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828822.199478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828822.199478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828822.455543:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828823.223421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828823.223421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828823.479448:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828824.247457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828824.247461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828824.503569:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828824.759498:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828824.759510:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828824.759515:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828824.759526:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5275 00000100:00080000:0.0:1787828824.759538:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828824.759546:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828824.759602:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973030016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828824.759650:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828824.759655:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828824.759657:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828824.759661:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5275 00000100:00080000:0.0:1787828824.759666:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828824.759671:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828824.759680:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828824.759695:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973030144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828824.759706:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828824.759709:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828824.759711:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828824.759716:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5275 00000100:00080000:0.0:1787828824.759721:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828824.759726:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828824.759732:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828824.759741:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973030016:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828824.759750:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973030272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828824.759786:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828824.759790:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828824.759792:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828824.759796:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5275 00000100:00080000:0.0:1787828824.759801:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828824.759806:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828824.759811:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828824.759826:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973030400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828824.759855:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828824.759855:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828824.759861:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828824.759863:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828824.759873:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5275 00000100:00080000:0.0:1787828824.759878:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828824.759881:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828824.759885:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828824.759897:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973030528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828824.759919:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828824/real 1787828824] req@ffff9c1a788621c0 x1874668973030016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828824.759925:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828824.759933:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828824.759936:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828824.759942:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5275 00000100:00080000:0.0:1787828824.759945:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828824.759949:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973030016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828824.759950:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828824.759959:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828824.759984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828824.759991:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973030656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828824.759991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828824.760000:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973030016:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828824.760016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973030144:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828824.760020:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828824.760038:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828824.760040:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828824.760061:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5275 00000100:00080000:0.0:1787828824.760065:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828824.760067:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828824.760072:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828824.760081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828824.760085:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973030784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828824.760099:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828824/real 1787828824] req@ffff9c1a78862940 x1874668973030144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828824.760122:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973030144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828824.760131:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828824.760135:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828824.760138:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973030144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828824.760189:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973030272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828824.760212:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828824.760241:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828824/real 1787828824] req@ffff9c1a78861a40 x1874668973030272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828824.760254:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973030272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828824.760263:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828824.760267:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828824.760271:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973030272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828824.760287:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973030400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828824.760307:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828824.760319:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828824/real 1787828824] req@ffff9c1a78860f00 x1874668973030400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828824.760331:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973030400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828824.760393:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828824.760396:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828824.760399:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973030400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828824.760404:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973030528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828824.760421:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828824.760429:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828824/real 1787828824] req@ffff9c1a78862d00 x1874668973030528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828824.760454:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973030528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828824.760461:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828824.760463:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828824.760465:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973030528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828824.760469:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973030656:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828824.760490:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828824.760497:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828824/real 1787828824] req@ffff9c1a78860b40 x1874668973030656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828824.760505:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973030656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828824.760511:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828824.760514:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828824.760516:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973030656:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828824.760521:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973030784:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828824.760533:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828824.760539:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828824/real 1787828824] req@ffff9c1a78863840 x1874668973030784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828824.760547:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973030784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828824.760556:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828824.760559:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828824.760563:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973030784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828825.271405:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828825.271429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828825.527545:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828826.295440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828826.295440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828826.551514:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828827.319445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828827.319445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828827.575479:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828828.343446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828828.343455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828828.599516:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828829.367479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828829.367522:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828829.623587:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828829.879547:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828829.879559:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828829.879575:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828829.879586:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5280 00000100:00080000:1.0:1787828829.879599:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828829.879608:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828829.879677:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973030912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828829.879717:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828829.879722:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828829.879724:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828829.879747:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5280 00000100:00080000:1.0:1787828829.879752:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828829.879759:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828829.879775:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828829.879785:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973030912:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828829.879801:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973031040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828829.879836:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828829.879840:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828829.879842:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828829.879846:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5280 00000100:00080000:1.0:1787828829.879851:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828829.879856:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828829.879863:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828829.879871:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828829.879906:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973031168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828829.879924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828829.879932:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828829/real 1787828829] req@ffff9c1a78a7b840 x1874668973030912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828829.879934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828829.879942:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828829.879958:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5280 00000100:00080000:1.0:1787828829.879971:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828829.879979:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828829.879985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973030912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828829.880010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828829.880014:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828829.880027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973030912:192.168.203.148@tcp:250:kworker.0 00000008:00100000:1.0:1787828829.880031:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828829.880047:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973031296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828829.880055:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973031040:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828829.880073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828829.880095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828829.880098:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828829.880107:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828829.880114:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5280 00000100:00080000:1.0:1787828829.880121:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828829.880133:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828829.880135:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828829/real 1787828829] req@ffff9c1a78a792c0 x1874668973031040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828829.880151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828829.880157:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973031040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828829.880167:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973031424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828829.880168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828829.880184:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828829.880200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973031040:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828829.880202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828829.880208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828829.880210:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828829.880215:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5280 00000100:00080000:1.0:1787828829.880228:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828829.880235:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828829.880235:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973031168:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787828829.880253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828829.880267:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973031552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828829.880275:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828829.880290:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828829.880291:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828829/real 1787828829] req@ffff9c1a78a79a40 x1874668973031168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828829.880329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828829.880336:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828829.880342:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973031168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828829.880345:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5280 00000100:00080000:1.0:1787828829.880352:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828829.880357:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828829.880362:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828829.880371:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973031680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828829.880384:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828829.880389:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828829.880392:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973031168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828829.880407:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973031296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828829.880433:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828829.880465:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828829/real 1787828829] req@ffff9c1a78a7b480 x1874668973031296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828829.880481:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973031296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828829.880489:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828829.880492:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828829.880495:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973031296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828829.880505:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973031424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828829.880527:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828829.880534:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828829/real 1787828829] req@ffff9c1a78a78f00 x1874668973031424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828829.880542:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973031424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828829.880547:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828829.880550:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828829.880552:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973031424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828829.880556:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973031552:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828829.880567:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828829.880573:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828829/real 1787828829] req@ffff9c1a78a7a580 x1874668973031552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828829.880580:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973031552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828829.880601:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828829.880603:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828829.880604:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973031552:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828829.880615:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973031680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828829.880633:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828829.880639:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828829/real 1787828829] req@ffff9c1a78a78000 x1874668973031680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828829.880646:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973031680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828829.880651:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828829.880654:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828829.880656:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973031680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828830.391439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828830.391441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828830.647484:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828831.415513:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828831.415513:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828831.671566:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828832.439450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828832.439451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828832.695550:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828833.463449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828833.463451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828833.719526:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828834.487447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828834.487448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828834.743472:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828834.999464:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828834.999472:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828834.999476:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828834.999486:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5285 00000100:00080000:0.0:1787828834.999505:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828834.999513:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828834.999549:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973031808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828834.999578:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828834.999581:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828834.999582:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828834.999586:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5285 00000100:00080000:0.0:1787828834.999590:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828834.999593:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828834.999600:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828834.999612:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973031936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828834.999620:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828834.999622:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828834.999624:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828834.999627:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5285 00000100:00080000:0.0:1787828834.999629:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828834.999631:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828834.999634:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828834.999643:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973032064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828834.999646:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973031808:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828834.999654:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828834.999657:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828834.999658:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828834.999660:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5285 00000100:00080000:0.0:1787828834.999662:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828834.999664:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828834.999668:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828834.999683:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973032192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828834.999697:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828834.999699:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828834.999701:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828834.999703:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5285 00000100:00080000:0.0:1787828834.999706:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828834.999708:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828834.999711:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828834.999718:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973032320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828834.999724:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828834.999726:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828834.999727:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828834.999730:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5285 00000100:00080000:0.0:1787828834.999732:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828834.999734:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828834.999737:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828834.999745:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973032448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828834.999755:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828834.999758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828834.999761:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828834.999765:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5285 00000100:00080000:0.0:1787828834.999771:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828834.999776:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828834.999782:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973031936:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787828834.999784:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828834.999803:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973032576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828834.999839:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973032064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828834.999872:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973032192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828834.999899:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973032320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828834.999923:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973032448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828834.999945:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973032576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828835.511476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828835.511506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828835.767538:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828836.535453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828836.535453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828836.791547:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828837.559423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828837.559423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828837.815527:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828838.071679:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787828838.071739:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828838.071761:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828838.071802:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828838.071809:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828838.071817:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828838.071830:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828838.071839:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787828838.071845:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787828838.071856:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828834/real 1787828838] req@ffff9c1a78860b40 x1874668973032064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.071886:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973032064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.071900:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.071906:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828838.071911:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973032064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828838.071920:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828834/real 1787828838] req@ffff9c1a78862d00 x1874668973032192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.071949:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973032192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.071960:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.071964:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828838.071969:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973032192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828838.071977:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828834/real 1787828838] req@ffff9c1a78860f00 x1874668973032320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.071991:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973032320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.072000:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.072004:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828838.072006:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973032320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828838.072012:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828834/real 1787828838] req@ffff9c1a78861a40 x1874668973032448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.072020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973032448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.072026:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.072028:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828838.072031:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973032448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828838.072035:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828834/real 1787828838] req@ffff9c1a78862940 x1874668973032576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.072042:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973032576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.072049:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.072052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828838.072056:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973032576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828838.072063:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828834/real 1787828838] req@ffff9c1a788612c0 x1874668973031808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.072076:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973031808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.072084:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.072089:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828838.072092:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973031808:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828838.072099:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828834/real 1787828838] req@ffff9c1a78863840 x1874668973031936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.072117:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973031936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.072126:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.072129:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828838.072133:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973031936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828838.583464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828838.583464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828838.839536:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828839.607454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828839.607481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828839.863583:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828840.119524:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828840.119535:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828840.119540:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828840.119553:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5290 00000100:00080000:2.0:1787828840.119582:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828840.119590:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828840.119640:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973032704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828840.119687:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828840.119690:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828840.119691:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828840.119695:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5290 00000100:00080000:2.0:1787828840.119697:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828840.119701:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828840.119715:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828840.119727:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973032832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828840.119736:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973032704:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828840.119738:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828840.119762:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828840.119765:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828840.119769:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5290 00000100:00080000:2.0:1787828840.119772:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828840.119775:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828840.119779:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828840.119787:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973032960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828840.119798:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828840.119803:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828840.119820:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828840.119824:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5290 00000100:00080000:2.0:1787828840.119829:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828840.119833:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828840.119838:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828840.119851:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973033088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828840.119861:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828840.119864:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828840.119866:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828840.119870:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5290 00000100:00080000:2.0:1787828840.119875:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828840.119876:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828840.119883:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828840.119891:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828840.119905:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973033216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828840.119916:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828840.119933:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828840.119935:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828840.119939:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5290 00000100:00080000:2.0:1787828840.119944:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828840.119949:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828840] req@ffff9c1a49d8e1c0 x1874668973032704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828840.119952:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828840.119961:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828840.119990:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973032704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828840.119996:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973033344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828840.120012:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828840.120014:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828840.120020:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787828840.120021:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828840.120027:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828840.120030:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973032704:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828840.120035:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5290 00000100:00080000:2.0:1787828840.120042:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828840.120048:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828840.120054:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828840.120068:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973033472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828840.120074:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973032832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828840.120110:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828840.120128:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828840] req@ffff9c1a49d8c3c0 x1874668973032832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828840.120142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973032832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828840.120150:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828840.120153:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828840.120157:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973032832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828840.120163:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973032960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828840.120185:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828840.120197:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828840] req@ffff9c1a49d8e580 x1874668973032960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828840.120210:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973032960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828840.120219:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828840.120222:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828840.120225:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973032960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828840.120232:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973033088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828840.120253:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828840.120264:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828840] req@ffff9c1a49d8fc00 x1874668973033088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828840.120274:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973033088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828840.120283:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828840.120286:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828840.120290:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973033088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828840.120295:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973033216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828840.120313:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828840.120326:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828840] req@ffff9c1a49d8ed00 x1874668973033216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828840.120337:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973033216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828840.120384:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828840.120389:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828840.120392:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973033216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828840.120399:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973033344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828840.120422:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828840.120436:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828840] req@ffff9c1a49d8d680 x1874668973033344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828840.120548:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973033344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828840.120554:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828840.120557:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828840.120559:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973033344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828840.120578:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973033472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828840.120596:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828840.120604:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828840/real 1787828840] req@ffff9c1a49d8f0c0 x1874668973033472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828840.120612:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973033472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828840.120618:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828840.120620:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828840.120622:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973033472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828840.631446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828840.631446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828840.887583:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828841.655482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828841.655482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828841.911515:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828842.679453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828842.679454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828842.935562:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828843.703458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828843.703472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828843.959583:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828844.727452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828844.727453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828844.983483:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828845.239512:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828845.239523:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828845.239528:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828845.239537:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5295 00000100:00080000:1.0:1787828845.239548:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828845.239556:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828845.239593:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973033600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828845.239625:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828845.239629:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828845.239631:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828845.239635:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5295 00000100:00080000:1.0:1787828845.239639:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828845.239643:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828845.239654:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828845.239685:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973033600:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828845.239697:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973033728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828845.239712:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828845.239722:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828845.239725:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828845.239730:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5295 00000100:00080000:1.0:1787828845.239735:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828845.239741:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787828845.239744:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:1.0:1787828845.239750:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828845.239783:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973033856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828845.239795:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828845.239796:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff9c1a78a78000 x1874668973033600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.239806:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828845.239810:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828845.239816:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5295 00000100:00100000:3.0:1787828845.239837:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973033600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.239840:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828845.239851:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787828845.239852:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828845.239860:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:1.0:1787828845.239863:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828845.239865:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973033600:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828845.239904:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973033984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828845.239908:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973033728:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828845.239926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828845.239931:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828845.239934:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828845.239936:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828845.239942:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5295 00000100:00080000:1.0:1787828845.239949:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828845.239951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff9c1a78a7a580 x1874668973033728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.239959:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828845.239969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973033728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828845.239971:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828845.239984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828845.239987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828845.239990:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973033728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828845.239996:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973034112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828845.239999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973033856:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828845.240016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828845.240021:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828845.240028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828845.240047:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828845.240048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff9c1a78a78f00 x1874668973033856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.240057:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5295 00000100:00100000:3.0:1787828845.240063:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973033856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.240065:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828845.240075:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787828845.240076:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828845.240083:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828845.240086:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973033856:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787828845.240088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828845.240097:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973033984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828845.240106:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973034240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828845.240116:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828845.240126:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828845.240128:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff9c1a78a7b480 x1874668973033984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.240135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828845.240140:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828845.240149:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5295 00000100:00100000:3.0:1787828845.240155:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973033984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828845.240156:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828845.240164:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828845.240176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828845.240177:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828845.240186:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828845.240190:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973033984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828845.240196:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973034368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828845.240201:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973034112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787828845.240243:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828845.240258:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff9c1a78a79a40 x1874668973034112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828845.240270:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973034112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828845.240278:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828845.240281:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828845.240284:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973034112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828845.240298:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973034240:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828845.240319:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828845.240331:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff9c1a78a792c0 x1874668973034240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828845.240381:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973034240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828845.240392:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828845.240395:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828845.240399:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973034240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787828845.240406:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973034368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787828845.240426:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828845.240438:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828845/real 1787828845] req@ffff9c1a78a7b840 x1874668973034368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828845.240451:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973034368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828845.240459:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828845.240462:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787828845.240466:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973034368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828845.751459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828845.751487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828846.007674:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828846.775421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828846.775453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828847.031564:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828847.799431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828847.799458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828848.055546:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828848.823463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828848.823492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828849.079483:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828849.847419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828849.847449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828850.103508:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828850.359463:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828850.359473:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828850.359479:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828850.359492:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5300 00000100:00080000:1.0:1787828850.359506:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828850.359518:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828850.359564:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973034496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828850.359596:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828850.359599:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828850.359601:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828850.359604:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5300 00000100:00080000:1.0:1787828850.359607:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828850.359611:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828850.359622:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828850.359633:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973034624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828850.359642:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828850.359648:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828850.359648:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973034496:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828850.359650:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828850.359657:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5300 00000100:00080000:1.0:1787828850.359660:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828850.359663:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828850.359667:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828850.359680:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973034752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828850.359693:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828850.359698:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828850.359701:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828850.359706:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5300 00000100:00080000:1.0:1787828850.359710:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828850.359716:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828850.359721:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828850.359728:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828850.359737:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973034880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828850.359746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828850.359749:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828850.359751:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828850.359754:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5300 00000100:00080000:1.0:1787828850.359757:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828850.359760:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828850.359764:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828850.359773:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973035008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828850.359780:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828850.359785:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828850.359785:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff9c1a78a7bc00 x1874668973034496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828850.359787:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828850.359795:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5300 00000100:00080000:1.0:1787828850.359800:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828850.359806:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828850.359811:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828850.359822:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973035136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828850.359827:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973034496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828850.359830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828850.359836:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828850.359838:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828850.359861:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5300 00000100:00080000:1.0:1787828850.359866:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828850.359868:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828850.359870:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828850.359875:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:1.0:1787828850.359876:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828850.359882:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973034496:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828850.359889:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973035264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828850.359914:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973034624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828850.359939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828850.359950:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff9c1a78a78780 x1874668973034624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828850.359959:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973034624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828850.359965:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828850.359968:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828850.359971:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973034624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828850.359975:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973034752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828850.359997:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828850.360006:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff9c1a78a7a940 x1874668973034752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828850.360015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973034752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828850.360021:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828850.360024:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828850.360027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973034752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828850.360033:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973034880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828850.360047:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828850.360058:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff9c1a78a783c0 x1874668973034880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828850.360067:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973034880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828850.360073:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828850.360076:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828850.360079:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973034880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828850.360086:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973035008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828850.360102:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828850.360110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff9c1a78a7a1c0 x1874668973035008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828850.360118:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973035008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828850.360124:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828850.360126:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828850.360128:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973035008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828850.360132:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973035136:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828850.360143:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828850.360150:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff9c1a78a7b0c0 x1874668973035136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828850.360157:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973035136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828850.360163:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828850.360164:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828850.360166:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973035136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828850.360170:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973035264:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828850.360179:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828850.360184:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828850/real 1787828850] req@ffff9c1a78a79e00 x1874668973035264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828850.360192:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973035264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828850.360197:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828850.360198:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828850.360200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973035264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828850.871488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828850.871503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828851.127542:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828851.895471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828851.895475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828852.151516:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828852.919458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828852.919480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828853.175538:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828853.943525:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828853.943532:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828854.199561:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828854.967474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828854.967491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828855.223570:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828855.479517:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828855.479530:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828855.479538:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828855.479554:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5305 00000100:00080000:1.0:1787828855.479581:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828855.479592:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828855.479672:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973035392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828855.479738:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828855.479745:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828855.479748:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828855.479755:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5305 00000100:00080000:1.0:1787828855.479761:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828855.479768:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828855.479795:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828855.479871:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973035392:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828855.479872:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973035520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828855.479911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828855.479916:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828855.479920:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828855.479927:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5305 00000100:00080000:1.0:1787828855.479932:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828855.479937:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828855.479945:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828855.479963:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973035648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828855.479984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828855.479988:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828855.479993:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828855.479995:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828855.480005:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5305 00000100:00080000:1.0:1787828855.480013:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828855.480021:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828855.480030:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828855.480055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973035776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828855.480074:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828855/real 1787828855] req@ffff9c1a78a79e00 x1874668973035392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828855.480163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828855.480166:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973035392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828855.480173:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828855.480184:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828855.480212:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5305 00000100:00080000:1.0:1787828855.480223:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828855.480225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828855.480239:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828855.480241:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828855.480267:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973035392:192.168.203.148@tcp:250:kworker.0 00000008:00100000:1.0:1787828855.480279:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828855.480301:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973035904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828855.480320:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828855.480326:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828855.480333:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828855.480333:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973035520:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828855.480423:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5305 00000100:00080000:1.0:1787828855.480433:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828855.480442:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787828855.480450:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:1.0:1787828855.480452:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828855.480491:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973036032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828855.480491:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828855/real 1787828855] req@ffff9c1a78a7b0c0 x1874668973035520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828855.480593:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828855.480600:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973035520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828855.480605:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828855.480617:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828855.480646:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5305 00000100:00080000:2.0:1787828855.480650:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828855.480672:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828855.480684:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828855.480686:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973035520:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828855.480706:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828855.480732:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973035648:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787828855.480735:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828855.480796:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973036160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828855.480841:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828855.480869:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828855/real 1787828855] req@ffff9c1a78a7a1c0 x1874668973035648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828855.480886:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973035648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828855.480902:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828855.480910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828855.480915:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973035648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828855.480924:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973035776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828855.480947:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828855.480957:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828855/real 1787828855] req@ffff9c1a78a783c0 x1874668973035776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828855.480967:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973035776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828855.480976:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828855.480979:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828855.480982:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973035776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828855.481002:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973035904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828855.481019:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828855.481029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828855/real 1787828855] req@ffff9c1a78a7a940 x1874668973035904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828855.481040:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973035904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828855.481049:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828855.481052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828855.481055:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973035904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828855.481062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973036032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828855.481078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828855.481087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828855/real 1787828855] req@ffff9c1a78a78780 x1874668973036032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828855.481098:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973036032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828855.481105:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828855.481108:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828855.481111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973036032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828855.481117:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973036160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828855.481131:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828855.481139:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828855/real 1787828855] req@ffff9c1a78a7bc00 x1874668973036160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828855.481150:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973036160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828855.481157:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828855.481160:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828855.481163:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973036160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828855.991417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828855.991467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828856.247572:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828857.015455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828857.015478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828857.271645:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828858.039440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828858.039457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828858.295528:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828859.063505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828859.063529:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828859.319428:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828860.087454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828860.087454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828860.343530:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828860.599434:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828860.599445:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828860.599449:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828860.599461:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5310 00000100:00080000:1.0:1787828860.599475:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828860.599483:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828860.599535:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973036288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828860.599565:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828860.599569:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828860.599571:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828860.599576:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5310 00000100:00080000:1.0:1787828860.599580:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828860.599586:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828860.599600:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828860.599614:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973036288:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828860.599620:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973036416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828860.599635:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828860.599639:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828860.599641:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828860.599646:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5310 00000100:00080000:1.0:1787828860.599651:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828860.599656:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828860.599663:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828860.599677:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973036544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828860.599691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828860.599691:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828860.599699:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828860.599701:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828860.599707:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5310 00000100:00080000:1.0:1787828860.599711:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828860.599716:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828860.599720:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828860.599732:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973036672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828860.599740:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828860.599741:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828860/real 1787828860] req@ffff9c1a78a7b840 x1874668973036288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828860.599748:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828860.599752:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828860.599762:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5310 00000100:00080000:1.0:1787828860.599765:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828860.599769:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828860.599772:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973036288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828860.599774:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828860.599802:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828860.599808:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973036800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828860.599809:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828860.599817:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973036288:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828860.599828:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828860.599833:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828860.599835:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828860.599841:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5310 00000100:00080000:1.0:1787828860.599845:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828860.599850:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973036416:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828860.599853:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828860.599864:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828860.599882:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973036928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828860.599882:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828860.599896:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828860.599899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828860/real 1787828860] req@ffff9c1a78a792c0 x1874668973036416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828860.599900:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828860.599904:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828860.599913:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5310 00000100:00080000:1.0:1787828860.599918:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828860.599920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973036416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828860.599925:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828860.599935:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828860.599937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828860.599943:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828860.599946:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973036416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828860.599949:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973037056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828860.599956:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973036544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828860.599978:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828860.599987:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828860/real 1787828860] req@ffff9c1a78a79a40 x1874668973036544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828860.599995:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973036544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828860.600000:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828860.600002:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828860.600004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973036544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828860.600008:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973036672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828860.600019:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828860.600025:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828860/real 1787828860] req@ffff9c1a78a7b480 x1874668973036672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828860.600032:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973036672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828860.600038:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828860.600040:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828860.600043:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973036672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828860.600049:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973036800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828860.600069:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828860.600081:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828860/real 1787828860] req@ffff9c1a78a78f00 x1874668973036800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828860.600102:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973036800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828860.600108:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828860.600111:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828860.600113:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973036800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828860.600128:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973036928:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828860.600145:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828860.600153:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828860/real 1787828860] req@ffff9c1a78a7a580 x1874668973036928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828860.600160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973036928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828860.600166:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828860.600168:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828860.600170:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973036928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828860.600173:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973037056:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828860.600184:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828860.600189:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828860/real 1787828860] req@ffff9c1a78a78000 x1874668973037056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828860.600197:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973037056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828860.600207:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828860.600209:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828860.600211:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973037056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828861.111482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828861.111482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828861.367511:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828862.135409:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828862.135418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828862.391606:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828863.159496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828863.159496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828863.415610:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828864.183505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828864.183537:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828864.439525:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828865.207411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828865.207411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828865.463499:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828865.719489:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828865.719501:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828865.719505:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828865.719516:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5316 00000100:00080000:1.0:1787828865.719525:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828865.719531:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828865.719574:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973037184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828865.719615:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828865.719618:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828865.719620:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828865.719624:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5316 00000100:00080000:1.0:1787828865.719628:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828865.719631:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828865.719640:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828865.719654:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973037312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828865.719664:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828865.719667:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828865.719668:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828865.719672:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5316 00000100:00080000:1.0:1787828865.719674:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828865.719677:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828865.719681:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828865.719694:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973037440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828865.719694:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973037184:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828865.719731:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828865.719733:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828865.719735:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828865.719788:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5316 00000100:00080000:1.0:1787828865.719791:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828865.719794:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828865.719798:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828865.719809:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973037568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828865.719816:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828865.719819:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828865.719820:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828865.719822:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5316 00000100:00080000:1.0:1787828865.719825:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828865.719827:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828865.719848:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828865.719855:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828865.719866:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973037696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828865.719875:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828865.719878:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828865.719879:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828865.719883:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5316 00000100:00080000:1.0:1787828865.719887:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828865.719891:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828865.719912:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828865/real 1787828865] req@ffff9c1a78a78000 x1874668973037184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828865.719915:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828865.719933:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973037824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828865.719940:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973037184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828865.719999:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828865.720000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828865.720005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828865.720006:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828865.720008:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828865.720013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973037184:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828865.720014:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5316 00000100:00080000:1.0:1787828865.720021:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828865.720026:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973037312:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828865.720027:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828865.720035:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828865.720071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973037952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828865.720078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828865.720110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828865/real 1787828865] req@ffff9c1a78a7a580 x1874668973037312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828865.720126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973037312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828865.720136:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828865.720140:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828865.720143:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973037312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828865.720167:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973037440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828865.720181:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828865.720190:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828865/real 1787828865] req@ffff9c1a78a78f00 x1874668973037440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828865.720200:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973037440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828865.720207:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828865.720209:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828865.720211:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973037440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828865.720232:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973037568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828865.720243:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828865.720250:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828865/real 1787828865] req@ffff9c1a78a7b480 x1874668973037568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828865.720257:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973037568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828865.720262:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828865.720264:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828865.720266:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973037568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828865.720270:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973037696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828865.720280:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828865.720286:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828865/real 1787828865] req@ffff9c1a78a79a40 x1874668973037696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828865.720293:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973037696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828865.720298:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828865.720300:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828865.720302:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973037696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828865.720324:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973037824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828865.720347:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828865.720361:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828865/real 1787828865] req@ffff9c1a78a792c0 x1874668973037824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828865.720377:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973037824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828865.720389:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828865.720393:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828865.720397:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973037824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828865.720405:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973037952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828865.720425:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828865.720435:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828865/real 1787828865] req@ffff9c1a78a7b840 x1874668973037952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828865.720445:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973037952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828865.720467:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828865.720498:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828865.720501:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973037952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828866.231535:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828866.231589:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828866.487471:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828867.255420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828867.255422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828867.511590:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828868.279465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828868.279465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828868.535473:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828869.303414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828869.303418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828869.559501:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828870.327422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828870.327532:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828870.583523:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828870.839441:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828870.839449:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828870.839453:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828870.839462:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5321 00000100:00080000:0.0:1787828870.839471:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828870.839477:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828870.839507:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973038080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828870.839546:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828870.839548:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828870.839550:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828870.839553:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5321 00000100:00080000:0.0:1787828870.839556:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828870.839559:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828870.839566:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828870.839576:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973038208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828870.839583:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828870.839586:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828870.839587:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828870.839589:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5321 00000100:00080000:0.0:1787828870.839591:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828870.839594:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828870.839596:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828870.839607:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973038336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828870.839607:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973038080:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828870.839623:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828870.839625:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828870.839626:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828870.839629:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5321 00000100:00080000:0.0:1787828870.839631:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828870.839633:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828870.839636:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828870.839645:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973038464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828870.839651:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828870.839654:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828870.839655:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828870.839657:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5321 00000100:00080000:0.0:1787828870.839660:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828870.839662:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828870.839665:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828870.839671:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828870.839674:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973038592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828870.839681:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828870.839683:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828870.839684:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828870.839687:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5321 00000100:00080000:0.0:1787828870.839690:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828870.839693:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828870.839697:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828870.839703:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973038720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828870.839726:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828870.839728:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828870.839733:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828870.839734:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828870/real 1787828870] req@ffff9c1a788621c0 x1874668973038080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828870.839737:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5321 00000100:00080000:0.0:1787828870.839742:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828870.839747:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828870.839754:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828870.839764:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973038080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828870.839767:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973038848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828870.839799:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828870.839803:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828870.839807:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973038080:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828870.839813:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973038208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828870.839835:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828870.839844:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828870/real 1787828870] req@ffff9c1a78860780 x1874668973038208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828870.839853:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973038208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828870.839859:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828870.839861:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828870.839863:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973038208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828870.839882:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973038336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828870.839893:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828870.839899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828870/real 1787828870] req@ffff9c1a78861e00 x1874668973038336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828870.839906:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973038336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828870.839912:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828870.839914:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828870.839916:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973038336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828870.839919:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973038464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828870.839934:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828870.839945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828870/real 1787828870] req@ffff9c1a78862580 x1874668973038464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828870.839965:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973038464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828870.839975:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828870.839978:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828870.839982:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973038464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828870.839988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973038592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828870.840005:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828870.840013:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828870/real 1787828870] req@ffff9c1a78861680 x1874668973038592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828870.840021:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973038592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828870.840027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828870.840029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828870.840031:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973038592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828870.840034:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973038720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828870.840045:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828870.840051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828870/real 1787828870] req@ffff9c1a788630c0 x1874668973038720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828870.840058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973038720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828870.840067:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828870.840068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828870.840070:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973038720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828870.840073:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973038848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828870.840083:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828870.840089:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828870/real 1787828870] req@ffff9c1a78863c00 x1874668973038848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828870.840095:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973038848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828870.840100:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828870.840102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828870.840104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973038848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828871.351396:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828871.351415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828871.607499:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828872.375468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828872.375468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828872.631476:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828873.399494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828873.399494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828873.655453:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828874.423462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828874.423464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828874.679570:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828875.447453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828875.447454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828875.703472:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828875.959468:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828875.959479:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828875.959483:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828875.959495:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5326 00000100:00080000:0.0:1787828875.959506:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828875.959514:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828875.959556:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973038976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828875.959589:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828875.959594:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828875.959596:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828875.959602:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5326 00000100:00080000:0.0:1787828875.959606:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828875.959611:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828875.959627:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828875.959645:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973039104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828875.959659:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828875.959663:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828875.959666:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828875.959688:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5326 00000100:00080000:0.0:1787828875.959693:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828875.959698:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973038976:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828875.959702:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828875.959729:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828875.959747:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973039232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828875.959763:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828875.959767:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828875.959770:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828875.959775:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5326 00000100:00080000:0.0:1787828875.959780:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828875.959787:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828875.959794:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828875.959798:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828875.959815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973039360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828875.959829:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828875.959834:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828875.959836:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828875.959841:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828875/real 1787828875] req@ffff9c1a78860000 x1874668973038976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828875.959846:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5326 00000100:00080000:0.0:1787828875.959855:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828875.959863:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828875.959874:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828875.959893:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973038976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828875.959895:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973039488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828875.959920:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828875.959926:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828875.959928:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828875.959936:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828875.959938:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973038976:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828875.959940:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828875.959948:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973039104:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828875.959951:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5326 00000100:00080000:0.0:1787828875.959959:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828875.959966:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828875.959974:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828875.959975:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828875.959990:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828875/real 1787828875] req@ffff9c1a78863c00 x1874668973039104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828875.959997:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973039616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828875.960008:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973039104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828875.960019:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828875.960022:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828875.960029:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828875.960029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828875.960035:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828875.960036:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973039104:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828875.960064:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5326 00000100:00080000:0.0:1787828875.960072:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828875.960077:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828875.960083:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973039232:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787828875.960085:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828875.960107:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973039744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828875.960112:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828875.960125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828875/real 1787828875] req@ffff9c1a788630c0 x1874668973039232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828875.960137:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973039232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828875.960149:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828875.960152:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828875.960155:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973039232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828875.960160:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973039360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828875.960171:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828875.960178:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828875/real 1787828875] req@ffff9c1a78861680 x1874668973039360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828875.960186:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973039360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828875.960192:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828875.960194:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828875.960196:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973039360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828875.960200:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973039488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828875.960211:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828875.960217:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828875/real 1787828875] req@ffff9c1a78862580 x1874668973039488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828875.960225:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973039488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828875.960230:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828875.960232:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828875.960235:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973039488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828875.960239:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973039616:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828875.960250:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828875.960256:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828875/real 1787828875] req@ffff9c1a78861e00 x1874668973039616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828875.960264:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973039616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828875.960270:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828875.960271:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828875.960273:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973039616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828875.960285:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973039744:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828875.960296:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828875.960302:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828875/real 1787828875] req@ffff9c1a78860780 x1874668973039744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828875.960310:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973039744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828875.960316:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828875.960317:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828875.960319:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973039744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828876.471448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828876.471448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828876.727554:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828877.495431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828877.495453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828877.751590:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828878.519450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828878.519450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828878.775523:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828879.543433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828879.543469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828879.799580:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828880.567500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828880.567504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828880.823485:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828881.079445:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828881.079453:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828881.079456:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828881.079465:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5331 00000100:00080000:0.0:1787828881.079474:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828881.079480:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828881.079512:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973039872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828881.079535:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828881.079538:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828881.079539:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828881.079542:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5331 00000100:00080000:0.0:1787828881.079545:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828881.079547:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828881.079554:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828881.079563:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973040000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828881.079570:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828881.079572:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828881.079574:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828881.079577:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5331 00000100:00080000:0.0:1787828881.079580:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828881.079585:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828881.079585:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973039872:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787828881.079592:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828881.079602:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973040128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828881.079610:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828881.079612:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828881.079614:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828881.079616:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5331 00000100:00080000:0.0:1787828881.079619:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828881.079621:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828881.079625:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828881.079632:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973040256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828881.079655:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828881.079657:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1787828881.079660:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828881.079661:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828881.079668:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5331 00000100:00080000:0.0:1787828881.079672:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828881.079675:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828881.079680:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828881.079692:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973040384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828881.079698:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828881.079702:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff9c1a788621c0 x1874668973039872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828881.079703:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828881.079706:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828881.079713:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5331 00000100:00080000:0.0:1787828881.079719:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828881.079726:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973039872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828881.079730:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828881.079741:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828881.079742:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828881.079749:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828881.079753:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973039872:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828881.079757:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973040512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828881.079765:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828881.079767:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828881.079769:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828881.079772:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973040000:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828881.079773:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5331 00000100:00080000:0.0:1787828881.079775:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828881.079779:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828881.079784:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828881.079794:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973040640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828881.079802:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828881.079813:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff9c1a78863840 x1874668973040000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828881.079824:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973040000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828881.079830:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828881.079833:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828881.079836:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973040000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828881.079855:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973040128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828881.079867:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828881.079873:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff9c1a788612c0 x1874668973040128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828881.079881:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973040128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828881.079886:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828881.079889:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828881.079891:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973040128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828881.079894:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973040256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828881.079903:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828881.079909:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff9c1a78862940 x1874668973040256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828881.079916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973040256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828881.079922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828881.079923:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828881.079925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973040256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828881.079929:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973040384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828881.079939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828881.079944:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff9c1a78861a40 x1874668973040384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828881.079956:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973040384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828881.079961:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828881.079963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828881.079965:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973040384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828881.079968:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973040512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828881.079978:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828881.079984:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff9c1a78860f00 x1874668973040512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828881.079991:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973040512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828881.079996:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828881.079998:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828881.079999:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973040512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828881.080003:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973040640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828881.080012:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828881.080018:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828881/real 1787828881] req@ffff9c1a78862d00 x1874668973040640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828881.080025:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973040640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828881.080034:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828881.080037:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828881.080039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973040640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828881.591410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828881.591448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828881.847521:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828882.615437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828882.615438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828882.871519:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828883.639400:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828883.639463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828883.895627:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828884.663431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828884.663431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828884.919649:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828885.687463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828885.687463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828885.943521:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828886.199452:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828886.199463:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828886.199468:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828886.199480:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5336 00000100:00080000:0.0:1787828886.199492:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828886.199501:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828886.199534:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973040768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828886.199558:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828886.199561:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828886.199563:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828886.199566:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5336 00000100:00080000:0.0:1787828886.199569:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828886.199573:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828886.199581:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828886.199593:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973040768:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828886.199596:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973040896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828886.199621:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828886.199624:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828886.199626:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828886.199629:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5336 00000100:00080000:0.0:1787828886.199633:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828886.199637:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828886.199642:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828886.199654:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973041024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828886.199667:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828886.199668:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828886.199673:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828886.199676:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828886.199681:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5336 00000100:00080000:0.0:1787828886.199684:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828886.199688:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828886.199691:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828886.199700:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973041152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828886.199707:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828886.199707:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff9c1a78860b40 x1874668973040768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828886.199714:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828886.199718:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828886.199730:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5336 00000100:00100000:1.0:1787828886.199735:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973040768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828886.199736:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828886.199743:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828886.199749:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828886.199760:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828886.199764:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973041280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828886.199771:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828886.199775:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828886.199778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828886.199782:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828886.199782:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973040768:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828886.199790:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5336 00000100:00080000:0.0:1787828886.199794:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828886.199796:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828886.199800:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828886.199807:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973041408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828886.199814:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973040896:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828886.199817:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828886.199823:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828886.199825:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828886.199828:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5336 00000100:00080000:0.0:1787828886.199830:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828886.199833:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828886.199836:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828886.199847:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828886.199848:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973041536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828886.199861:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff9c1a78862d00 x1874668973040896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.199871:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973040896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.199878:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.199880:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828886.199882:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973040896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828886.199886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973041024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828886.199898:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828886.199904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff9c1a78860f00 x1874668973041024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.199916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973041024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.199922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.199923:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828886.199925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973041024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828886.199929:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973041152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828886.199939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828886.199945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff9c1a78861a40 x1874668973041152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.199952:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973041152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.199957:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.199959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828886.199961:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973041152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828886.199965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973041280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828886.199976:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828886.199981:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff9c1a78862940 x1874668973041280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.199988:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973041280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.200001:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.200005:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828886.200008:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973041280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828886.200028:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973041408:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828886.200053:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828886.200066:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff9c1a788612c0 x1874668973041408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.200075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973041408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.200082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.200085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828886.200087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973041408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828886.200092:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973041536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828886.200112:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828886.200126:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828886/real 1787828886] req@ffff9c1a78863840 x1874668973041536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828886.200137:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973041536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828886.200142:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828886.200144:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828886.200147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973041536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828886.711452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828886.711485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828886.967508:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828887.735433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828887.735433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828887.991489:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828888.759437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828888.759470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828889.015551:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828889.783466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828889.783487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828890.039606:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828890.807457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828890.807468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828891.063577:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828891.319535:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828891.319547:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828891.319552:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828891.319574:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5341 00000100:00080000:1.0:1787828891.319587:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828891.319595:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828891.319643:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973041664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828891.319674:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828891.319678:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828891.319679:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828891.319682:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5341 00000100:00080000:1.0:1787828891.319684:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828891.319687:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828891.319697:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828891.319705:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973041792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828891.319718:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828891.319721:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828891.319723:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828891.319728:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5341 00000100:00080000:1.0:1787828891.319731:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828891.319734:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828891.319738:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828891.319744:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973041920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828891.319755:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828891.319758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828891.319759:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828891.319764:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5341 00000100:00080000:1.0:1787828891.319767:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828891.319774:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828891.319779:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828891.319779:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973041664:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828891.319789:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973042048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828891.319806:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828891.319809:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828891.319811:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828891.319814:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5341 00000100:00080000:1.0:1787828891.319832:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828891.319838:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828891.319842:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828891.319853:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973042176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828891.319859:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828891.319862:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828891.319863:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828891.319867:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5341 00000100:00080000:1.0:1787828891.319871:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828891.319874:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828891.319878:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828891.319884:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973042304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828891.319890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828891.319893:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828891.319895:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828891.319901:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5341 00000100:00080000:1.0:1787828891.319906:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828891.319911:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828891.319917:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828891.319922:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828891.319931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973042432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828891.319981:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff9c1a78a7bc00 x1874668973041664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828891.320011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973041664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828891.320028:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828891.320035:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828891.320042:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973041664:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828891.320054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973041792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828891.320091:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828891.320133:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff9c1a78a78780 x1874668973041792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828891.320155:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973041792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828891.320168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828891.320180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828891.320186:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973041792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828891.320234:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973041920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828891.320272:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828891.320290:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff9c1a78a7a940 x1874668973041920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828891.320308:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973041920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828891.320320:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828891.320325:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828891.320331:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973041920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828891.320341:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973042048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828891.320415:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828891.320440:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff9c1a78a783c0 x1874668973042048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828891.320468:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973042048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828891.320481:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828891.320486:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828891.320491:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973042048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828891.320501:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973042176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828891.320530:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828891.320546:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff9c1a78a7a1c0 x1874668973042176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828891.320589:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973042176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828891.320603:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828891.320609:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828891.320614:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973042176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828891.320633:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973042304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828891.320666:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828891.320685:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff9c1a78a7b0c0 x1874668973042304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828891.320703:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973042304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828891.320726:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828891.320732:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828891.320737:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973042304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828891.320747:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973042432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828891.320775:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828891.320791:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828891/real 1787828891] req@ffff9c1a78a79e00 x1874668973042432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828891.320819:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973042432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828891.320831:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828891.320836:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828891.320841:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973042432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828891.831520:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828891.831525:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828892.087625:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828892.855448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828892.855450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828893.111523:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828893.879435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828893.879449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828894.135622:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828894.903438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828894.903440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828895.159513:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828895.927514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828895.927514:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828896.183599:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828896.439549:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828896.439561:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828896.439567:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828896.439580:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5346 00000100:00080000:0.0:1787828896.439595:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828896.439604:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828896.439655:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973042560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828896.439689:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828896.439694:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828896.439696:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828896.439700:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5346 00000100:00080000:0.0:1787828896.439704:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828896.439707:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828896.439719:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828896.439730:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973042688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828896.439739:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828896.439744:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828896.439746:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828896.439750:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5346 00000100:00080000:0.0:1787828896.439752:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828896.439757:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828896.439762:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828896.439769:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973042816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828896.439781:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828896.439786:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828896.439787:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828896.439795:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5346 00000100:00100000:2.0:1787828896.439797:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973042560:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828896.439800:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828896.439814:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828896.439819:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828896.439835:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973042944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828896.439844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828896.439846:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828896.439848:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828896.439852:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5346 00000100:00080000:0.0:1787828896.439859:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828896.439863:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828896.439868:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828896.439878:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973043072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828896.439890:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828896.439904:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828896.439909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828896.439912:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828896.439918:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5346 00000100:00080000:0.0:1787828896.439921:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828896.439925:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828896.439931:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828896.439946:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973043200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828896.439948:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff9c1a788621c0 x1874668973042560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828896.440004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828896.440018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828896.440022:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828896.440041:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5346 00000100:00080000:0.0:1787828896.440051:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828896.440057:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828896.440061:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828896.440064:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973042560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828896.440074:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973043328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828896.440076:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828896.440096:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828896.440106:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973042560:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828896.440117:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973042688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828896.440143:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828896.440153:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff9c1a78860780 x1874668973042688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828896.440162:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973042688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828896.440168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828896.440170:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828896.440172:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973042688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828896.440194:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973042816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828896.440206:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828896.440213:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff9c1a78861e00 x1874668973042816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828896.440221:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973042816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828896.440231:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828896.440233:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828896.440235:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973042816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828896.440239:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973042944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828896.440249:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828896.440255:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff9c1a78862580 x1874668973042944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828896.440262:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973042944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828896.440268:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828896.440270:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828896.440272:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973042944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828896.440276:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973043072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828896.440286:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828896.440292:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff9c1a78861680 x1874668973043072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828896.440300:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973043072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828896.440305:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828896.440307:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828896.440309:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973043072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828896.440313:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973043200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828896.440324:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828896.440329:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff9c1a788630c0 x1874668973043200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828896.440336:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973043200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828896.440340:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828896.440342:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828896.440378:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973043200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828896.440382:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973043328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828896.440396:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828896.440403:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828896/real 1787828896] req@ffff9c1a78863c00 x1874668973043328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828896.440411:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973043328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828896.440417:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828896.440419:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828896.440422:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973043328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828896.951420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828896.951420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828897.207545:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828897.975436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828897.975436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828898.231544:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828898.999425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828898.999425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828899.255491:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828900.023435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828900.023435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828900.279571:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828901.047434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828901.047436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828901.303519:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828901.559424:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828901.559433:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828901.559437:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828901.559446:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5351 00000100:00080000:2.0:1787828901.559457:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828901.559464:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828901.559511:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973043456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828901.559537:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828901.559539:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828901.559541:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828901.559544:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5351 00000100:00080000:2.0:1787828901.559548:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828901.559552:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828901.559560:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828901.559570:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973043456:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828901.559573:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973043584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828901.559584:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828901.559586:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828901.559588:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828901.559590:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5351 00000100:00080000:2.0:1787828901.559592:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828901.559595:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828901.559599:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828901.559607:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973043712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828901.559614:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828901.559616:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828901.559618:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828901.559620:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5351 00000100:00080000:2.0:1787828901.559629:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828901.559631:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828901.559634:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828901.559641:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973043840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828901.559647:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828901.559649:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828901.559650:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828901.559653:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5351 00000100:00080000:2.0:1787828901.559655:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828901.559658:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828901.559663:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828901.559670:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973043968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828901.559675:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828901.559678:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828901.559679:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828901.559682:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5351 00000100:00100000:1.0:1787828901.559686:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973043584:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828901.559687:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828901.559692:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828901.559699:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828901.559709:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973044096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828901.559717:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828901.559720:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828901.559722:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828901.559724:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5351 00000100:00100000:1.0:1787828901.559728:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973043712:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828901.559729:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828901.559735:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828901.559739:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828901.559749:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973044224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828901.559756:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973043840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828901.559775:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973043968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828901.559789:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973044096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828901.559804:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973044224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828902.071443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828902.071443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828902.327480:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828903.095422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828903.095430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828903.351516:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828904.119448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828904.119498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828904.375575:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828904.631783:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:3.0:1787828904.631863:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828904.631895:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787828904.631946:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787828904.631956:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787828904.631965:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787828904.631979:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787828904.631995:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787828904.632012:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787828904.632026:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828901/real 1787828904] req@ffff9c1a49d8de00 x1874668973043584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828904.632072:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973043584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828904.632092:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828904.632101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828904.632108:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973043584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828904.632121:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828901/real 1787828904] req@ffff9c1a49d8c780 x1874668973043712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828904.632142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973043712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828904.632155:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828904.632162:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828904.632168:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973043712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828904.632179:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828901/real 1787828904] req@ffff9c1a49d8c000 x1874668973043840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828904.632199:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973043840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828904.632214:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828904.632221:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828904.632227:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973043840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828904.632239:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828901/real 1787828904] req@ffff9c1a49d8cf00 x1874668973043968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828904.632260:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973043968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828904.632273:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828904.632279:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828904.632285:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973043968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828904.632297:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828901/real 1787828904] req@ffff9c1a49d8cb40 x1874668973043456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828904.632318:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973043456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828904.632331:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828904.632338:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828904.632396:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973043456:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828904.632451:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828901/real 1787828904] req@ffff9c1a49d8da40 x1874668973044096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828904.632495:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973044096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828904.632510:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828904.632518:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828904.632524:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973044096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828904.632537:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828901/real 1787828904] req@ffff9c1a49d8f0c0 x1874668973044224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828904.632562:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973044224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828904.632577:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828904.632583:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828904.632588:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973044224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828905.143546:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828905.143560:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828905.399534:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828906.167430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828906.167430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828906.423514:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828906.679450:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828906.679470:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828906.679474:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828906.679483:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5357 00000100:00080000:0.0:1787828906.679492:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828906.679498:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828906.679529:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973044352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828906.679552:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828906.679555:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828906.679557:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828906.679560:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5357 00000100:00080000:0.0:1787828906.679563:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828906.679565:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828906.679573:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828906.679584:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973044480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828906.679592:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828906.679594:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828906.679596:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828906.679598:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5357 00000100:00080000:0.0:1787828906.679601:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828906.679605:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828906.679609:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973044352:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787828906.679610:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828906.679620:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973044608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828906.679631:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828906.679635:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828906.679638:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828906.679642:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5357 00000100:00080000:0.0:1787828906.679646:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828906.679651:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828906.679656:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828906.679670:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973044736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828906.679681:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828906.679682:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828906.679689:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828906.679691:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828906.679697:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5357 00000100:00080000:0.0:1787828906.679703:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828906.679707:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828906.679713:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828906.679729:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828906/real 1787828906] req@ffff9c1a78860000 x1874668973044352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828906.679736:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973044864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828906.679757:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828906.679760:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828906.679762:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828906.679765:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973044352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828906.679770:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5357 00000100:00080000:0.0:1787828906.679785:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828906.679791:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828906.679792:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828906.679802:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828906.679802:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828906.679810:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973044352:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828906.679818:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973044992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828906.679823:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973044480:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828906.679842:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828906.679846:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828906.679848:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828906.679853:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5357 00000100:00080000:0.0:1787828906.679861:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828906.679868:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787828906.679868:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787828906.679887:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828906.679898:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828906/real 1787828906] req@ffff9c1a78863c00 x1874668973044480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828906.679903:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973045120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828906.679930:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973044480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828906.679941:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828906.679945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828906.679951:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973044480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828906.679978:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973044608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828906.680002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828906.680013:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828906/real 1787828906] req@ffff9c1a788630c0 x1874668973044608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828906.680025:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973044608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828906.680034:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828906.680037:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828906.680040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973044608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828906.680047:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973044736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828906.680067:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828906.680079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828906/real 1787828906] req@ffff9c1a78861680 x1874668973044736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828906.680104:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973044736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828906.680113:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828906.680116:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828906.680119:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973044736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828906.680125:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973044864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828906.680144:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828906.680155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828906/real 1787828906] req@ffff9c1a78862580 x1874668973044864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828906.680165:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973044864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828906.680173:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828906.680175:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828906.680178:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973044864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828906.680184:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973044992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828906.680203:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828906.680213:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828906/real 1787828906] req@ffff9c1a78861e00 x1874668973044992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828906.680224:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973044992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828906.680238:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828906.680241:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828906.680245:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973044992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828906.680251:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973045120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828906.680270:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828906.680282:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828906/real 1787828906] req@ffff9c1a78860780 x1874668973045120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828906.680295:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973045120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828906.680305:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828906.680308:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828906.680311:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973045120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828907.191503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828907.191549:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828907.447481:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828908.215476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828908.215504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828908.471532:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828909.239526:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828909.239526:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828909.495423:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828910.263421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828910.263428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828910.519556:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828911.287472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828911.287472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828911.543546:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828911.799459:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828911.799473:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828911.799480:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828911.799494:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5362 00000100:00080000:1.0:1787828911.799508:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828911.799517:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828911.799564:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973045248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828911.799595:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828911.799597:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828911.799599:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828911.799602:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5362 00000100:00080000:1.0:1787828911.799605:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828911.799607:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828911.799617:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828911.799627:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973045376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828911.799635:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828911.799637:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828911.799639:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828911.799643:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5362 00000100:00080000:1.0:1787828911.799646:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828911.799648:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973045248:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828911.799650:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828911.799655:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828911.799663:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973045504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828911.799675:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828911.799678:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828911.799679:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828911.799681:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5362 00000100:00080000:1.0:1787828911.799684:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828911.799686:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828911.799689:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828911.799696:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973045632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828911.799702:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828911.799704:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828911.799705:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828911.799708:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5362 00000100:00080000:1.0:1787828911.799711:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828911.799714:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828911.799716:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828911.799722:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973045760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828911.799728:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828911.799730:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828911.799731:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828911.799734:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5362 00000100:00080000:1.0:1787828911.799739:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828911.799740:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828911.799744:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828911.799749:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828911.799756:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973045888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828911.799762:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828911.799765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828911.799766:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828911.799768:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5362 00000100:00080000:1.0:1787828911.799771:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828911.799773:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828911.799776:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828911.799784:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973046016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828911.799791:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828911/real 1787828911] req@ffff9c1a78a79e00 x1874668973045248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828911.799808:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973045248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828911.799818:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828911.799830:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828911.799833:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973045248:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828911.799857:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973045376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828911.799875:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828911.799885:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828911/real 1787828911] req@ffff9c1a78a7b0c0 x1874668973045376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828911.799893:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973045376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828911.799899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828911.799901:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828911.799903:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973045376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828911.799907:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973045504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828911.799919:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828911.799925:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828911/real 1787828911] req@ffff9c1a78a7a1c0 x1874668973045504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828911.799932:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973045504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828911.799937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828911.799939:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828911.799941:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973045504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828911.799945:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973045632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828911.799955:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828911.799961:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828911/real 1787828911] req@ffff9c1a78a783c0 x1874668973045632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828911.799968:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973045632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828911.799974:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828911.799976:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828911.799977:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973045632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828911.799981:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973045760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828911.799990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828911.799996:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828911/real 1787828911] req@ffff9c1a78a7a940 x1874668973045760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828911.800008:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973045760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828911.800013:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828911.800015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828911.800016:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973045760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828911.800020:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973045888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828911.800031:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828911.800037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828911/real 1787828911] req@ffff9c1a78a78780 x1874668973045888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828911.800044:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973045888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828911.800049:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828911.800051:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828911.800053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973045888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828911.800056:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973046016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828911.800065:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828911.800070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828911/real 1787828911] req@ffff9c1a78a7bc00 x1874668973046016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828911.800077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973046016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828911.800086:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828911.800088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828911.800090:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973046016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828912.311455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828912.311470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828912.567526:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828913.335489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828913.335498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828913.591590:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828914.359447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828914.359449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828914.615506:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828915.383482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828915.383482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828915.639477:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828916.407481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828916.407481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828916.663431:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828916.919427:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828916.919436:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828916.919439:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828916.919447:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5367 00000100:00080000:1.0:1787828916.919457:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828916.919463:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828916.919494:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973046144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828916.919518:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828916.919521:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828916.919522:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828916.919525:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5367 00000100:00080000:1.0:1787828916.919528:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828916.919530:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828916.919537:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828916.919546:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973046272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828916.919555:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828916.919557:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828916.919559:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828916.919561:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5367 00000100:00080000:1.0:1787828916.919563:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828916.919565:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828916.919568:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828916.919575:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973046144:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828916.919577:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973046400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828916.919589:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828916.919591:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828916.919592:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828916.919595:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5367 00000100:00080000:1.0:1787828916.919597:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828916.919600:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828916.919603:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828916.919612:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973046528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828916.919618:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828916.919620:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828916.919621:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828916.919623:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5367 00000100:00080000:1.0:1787828916.919626:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828916.919628:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828916.919631:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828916.919641:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973046656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828916.919642:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787828916.919650:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828916.919653:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828916.919655:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828916.919658:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5367 00000100:00080000:1.0:1787828916.919660:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828916.919663:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828916.919667:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828916.919674:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973046784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828916.919680:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828916.919684:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828916.919684:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828916/real 1787828916] req@ffff9c1a78a7b840 x1874668973046144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828916.919688:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828916.919695:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5367 00000100:00080000:1.0:1787828916.919701:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828916.919706:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828916.919709:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828916.919710:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973046144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828916.919719:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973046912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828916.919720:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828916.919729:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828916.919735:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973046144:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828916.919742:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973046272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828916.919758:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828916.919766:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828916/real 1787828916] req@ffff9c1a78a792c0 x1874668973046272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828916.919779:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973046272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828916.919785:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828916.919786:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828916.919788:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973046272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828916.919808:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973046400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828916.919819:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828916.919825:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828916/real 1787828916] req@ffff9c1a78a79a40 x1874668973046400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828916.919833:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973046400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828916.919838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828916.919841:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828916.919843:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973046400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828916.919846:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973046528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828916.919855:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828916.919861:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828916/real 1787828916] req@ffff9c1a78a7b480 x1874668973046528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828916.919868:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973046528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828916.919878:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828916.919880:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828916.919882:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973046528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828916.919886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973046656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828916.919895:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828916.919900:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828916/real 1787828916] req@ffff9c1a78a78f00 x1874668973046656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828916.919907:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973046656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828916.919913:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828916.919914:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828916.919916:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973046656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828916.919920:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973046784:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828916.919930:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828916.919936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828916/real 1787828916] req@ffff9c1a78a7a580 x1874668973046784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828916.919944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973046784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828916.919949:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828916.919950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828916.919953:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973046784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828916.919956:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973046912:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828916.919966:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828916.919971:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828916/real 1787828916] req@ffff9c1a78a78000 x1874668973046912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828916.919978:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973046912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828916.919984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828916.919986:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828916.919987:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973046912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828917.431461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828917.431461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828917.687491:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828918.455448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828918.455487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828918.711477:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828919.479444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828919.479444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828919.735490:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828920.503468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828920.503469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828920.759637:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828921.527455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828921.527459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828921.783622:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828922.039525:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828922.039536:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828922.039541:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828922.039555:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5372 00000100:00080000:1.0:1787828922.039567:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828922.039574:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828922.039622:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973047040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828922.039664:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828922.039668:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828922.039670:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828922.039674:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5372 00000100:00080000:1.0:1787828922.039676:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828922.039679:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828922.039690:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828922.039701:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973047168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828922.039711:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828922.039713:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828922.039714:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828922.039717:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5372 00000100:00080000:1.0:1787828922.039721:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828922.039731:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973047040:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828922.039743:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828922.039754:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828922.039772:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973047296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828922.039785:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828922.039790:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828922.039792:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828922.039798:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5372 00000100:00080000:1.0:1787828922.039815:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828922.039822:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828922.039828:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828922.039837:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828922.039848:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973047424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828922.039858:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828922.039861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828922.039863:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828922.039869:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5372 00000100:00080000:1.0:1787828922.039872:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828922.039876:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828922.039881:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828922.039897:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973047552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828922.039907:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff9c1a78a78000 x1874668973047040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828922.039908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828922.039923:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828922.039926:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828922.039942:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5372 00000100:00080000:1.0:1787828922.039946:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828922.039951:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828922.039967:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828922.039980:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973047680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828922.039989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828922.039990:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973047040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828922.039996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828922.039999:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828922.040008:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5372 00000100:00080000:1.0:1787828922.040015:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828922.040025:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828922.040033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828922.040037:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828922.040045:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828922.040047:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973047808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828922.040054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973047040:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828922.040106:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973047168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828922.040134:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828922.040149:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff9c1a78a7a580 x1874668973047168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828922.040158:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973047168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828922.040165:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828922.040184:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828922.040190:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973047168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828922.040199:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973047296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828922.040221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828922.040262:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff9c1a78a78f00 x1874668973047296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828922.040291:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973047296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828922.040307:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828922.040314:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828922.040322:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973047296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828922.040334:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973047424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828922.040431:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828922.040445:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff9c1a78a7b480 x1874668973047424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828922.040457:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973047424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828922.040466:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828922.040470:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828922.040474:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973047424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828922.040480:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973047552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828922.040499:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828922.040507:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff9c1a78a79a40 x1874668973047552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828922.040520:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973047552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828922.040526:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828922.040528:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828922.040546:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973047552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828922.040552:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973047680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828922.040568:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828922.040575:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff9c1a78a792c0 x1874668973047680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828922.040584:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973047680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828922.040590:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828922.040592:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828922.040594:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973047680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828922.040617:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973047808:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828922.040643:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828922.040659:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828922/real 1787828922] req@ffff9c1a78a7b840 x1874668973047808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828922.040676:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973047808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828922.040700:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828922.040703:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828922.040706:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973047808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828922.551475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828922.551493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828922.807515:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828923.575469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828923.575484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828923.831512:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828924.599473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828924.599473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828924.855520:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828925.623527:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828925.623534:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828925.879503:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828926.647465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828926.647465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828926.903575:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828927.159487:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828927.159502:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828927.159507:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828927.159521:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5377 00000100:00080000:0.0:1787828927.159536:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828927.159546:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828927.159598:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973047936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828927.159627:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828927.159630:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828927.159632:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828927.159635:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5377 00000100:00080000:0.0:1787828927.159639:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828927.159642:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828927.159651:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828927.159662:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973048064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828927.159675:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828927.159679:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828927.159683:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828927.159688:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5377 00000100:00080000:0.0:1787828927.159692:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828927.159697:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828927.159704:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828927.159717:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973047936:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828927.159718:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973048192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828927.159742:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828927.159744:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828927.159747:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828927.159769:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5377 00000100:00080000:0.0:1787828927.159773:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828927.159779:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828927.159786:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828927.159801:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973048320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828927.159813:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828927.159846:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828927.159849:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828927.159851:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828927.159856:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5377 00000100:00080000:0.0:1787828927.159861:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828927.159866:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828927.159871:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff9c1a788621c0 x1874668973047936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828927.159874:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828927.159896:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973048448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828927.159914:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828927.159916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973047936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828927.159920:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828927.159925:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828927.159948:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5377 00000100:00080000:0.0:1787828927.159954:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828927.159955:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828927.159961:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787828927.159968:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787828927.159970:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828927.159976:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973047936:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828927.159984:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973048576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828927.159988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973048064:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828927.160011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828927.160015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828927.160017:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828927.160021:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5377 00000100:00080000:0.0:1787828927.160024:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828927.160032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828927.160037:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828927.160042:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828927.160050:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff9c1a78863840 x1874668973048064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828927.160058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973048704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828927.160075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973048064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.160083:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.160086:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828927.160089:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973048064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828927.160115:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973048192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828927.160128:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828927.160134:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff9c1a788612c0 x1874668973048192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828927.160142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973048192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.160152:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.160154:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828927.160156:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973048192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828927.160160:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973048320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828927.160170:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828927.160175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff9c1a78862940 x1874668973048320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828927.160183:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973048320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.160188:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.160190:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828927.160192:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973048320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828927.160196:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973048448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828927.160206:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828927.160212:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff9c1a78861a40 x1874668973048448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828927.160219:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973048448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.160224:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.160226:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828927.160229:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973048448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828927.160233:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973048576:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828927.160244:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828927.160250:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff9c1a78860f00 x1874668973048576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828927.160257:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973048576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.160262:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.160264:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828927.160265:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973048576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828927.160269:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973048704:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828927.160279:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828927.160284:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828927/real 1787828927] req@ffff9c1a78862d00 x1874668973048704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828927.160292:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973048704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828927.160297:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828927.160299:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828927.160300:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973048704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828927.671493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828927.671494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828927.927544:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828928.695456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828928.695456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828928.951530:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828929.719473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828929.719473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828929.975568:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828930.743430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828930.743430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828930.999480:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828931.767437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828931.767451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828932.023500:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828932.279501:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.279524:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828932.279529:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828932.279540:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5382 00000100:00080000:2.0:1787828932.279553:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.279560:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828932.279614:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973048832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828932.279654:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.279657:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828932.279658:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828932.279662:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5382 00000100:00080000:2.0:1787828932.279664:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.279667:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828932.279677:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828932.279688:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973048960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828932.279698:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.279701:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828932.279702:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828932.279705:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5382 00000100:00080000:2.0:1787828932.279708:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.279717:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828932.279720:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973048832:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787828932.279723:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828932.279742:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973049088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828932.279754:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.279757:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828932.279759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828932.279763:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5382 00000100:00080000:2.0:1787828932.279766:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.279770:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828932.279774:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828932.279783:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973049216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828932.279794:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.279797:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828932.279799:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828932.279807:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5382 00000800:00000100:0.0:1787828932.279814:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828932.279814:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.279833:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828932.279838:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828932.279848:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973049344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828932.279861:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.279863:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828932.279865:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828932.279868:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5382 00000100:00080000:2.0:1787828932.279871:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.279874:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828932.279878:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828932.279889:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff9c1a49d8d680 x1874668973048832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828932.279891:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973049472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828932.279926:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828932.279929:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828932.279931:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828932.279935:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973048832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828932.279942:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5382 00000100:00080000:2.0:1787828932.279952:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828932.279965:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787828932.279967:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828932.279971:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828932.279975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828932.279980:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973048832:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828932.279988:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973049600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828932.280010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973048960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828932.280034:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828932.280043:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff9c1a49d8ed00 x1874668973048960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828932.280052:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973048960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828932.280057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828932.280060:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828932.280062:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973048960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828932.280067:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973049088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828932.280078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828932.280084:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff9c1a49d8fc00 x1874668973049088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828932.280091:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973049088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828932.280101:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828932.280103:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828932.280105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973049088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828932.280109:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973049216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828932.280119:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828932.280125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff9c1a49d8e580 x1874668973049216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828932.280132:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973049216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828932.280137:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828932.280139:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828932.280141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973049216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828932.280145:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973049344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828932.280155:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828932.280161:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff9c1a49d8c3c0 x1874668973049344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828932.280168:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973049344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828932.280173:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828932.280175:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828932.280177:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973049344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828932.280181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973049472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828932.280193:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828932.280199:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff9c1a49d8e1c0 x1874668973049472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828932.280206:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973049472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828932.280211:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828932.280213:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828932.280214:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973049472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828932.280218:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973049600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828932.280229:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828932.280234:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828932/real 1787828932] req@ffff9c1a49d8f0c0 x1874668973049600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828932.280241:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973049600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828932.280246:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828932.280248:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828932.280250:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973049600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828932.791461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828932.791462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828933.047578:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828933.815458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828933.815471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828934.071550:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828934.839499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828934.839500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828935.095527:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828935.863435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828935.863508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828936.119553:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828936.887485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828936.887488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828937.143521:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828937.399433:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828937.399441:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828937.399445:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828937.399453:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5387 00000100:00080000:1.0:1787828937.399462:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828937.399468:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828937.399500:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973049728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828937.399521:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828937.399525:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828937.399527:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828937.399531:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5387 00000100:00080000:1.0:1787828937.399535:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828937.399539:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828937.399549:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828937.399562:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973049856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828937.399569:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973049728:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787828937.399572:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828937.399575:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828937.399577:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828937.399581:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5387 00000100:00080000:1.0:1787828937.399584:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828937.399587:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828937.399592:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828937.399600:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973049984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828937.399609:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828937.399612:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828937.399613:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828937.399616:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5387 00000100:00080000:1.0:1787828937.399619:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828937.399622:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828937.399626:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828937.399631:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828937.399635:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973050112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828937.399644:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828937.399647:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828937.399649:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828937.399652:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5387 00000100:00080000:1.0:1787828937.399655:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828937.399659:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828937.399662:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828937.399672:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973050240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828937.399677:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff9c1a78a7bc00 x1874668973049728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828937.399682:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828937.399685:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828937.399687:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828937.399694:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973049728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828937.399700:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5387 00000100:00080000:1.0:1787828937.399703:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828937.399705:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828937.399707:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787828937.399709:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:1.0:1787828937.399712:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828937.399713:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973049728:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828937.399723:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973050368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828937.399731:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828937.399734:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973049856:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787828937.399734:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828937.399736:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828937.399740:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5387 00000100:00080000:1.0:1787828937.399743:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828937.399746:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828937.399750:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828937.399757:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828937.399760:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973050496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828937.399769:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff9c1a78a78780 x1874668973049856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828937.399781:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973049856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828937.399790:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828937.399793:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828937.399797:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973049856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828937.399803:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973049984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828937.399822:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828937.399831:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff9c1a78a7a940 x1874668973049984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828937.399843:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973049984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828937.399848:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828937.399850:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828937.399852:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973049984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828937.399856:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973050112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828937.399866:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828937.399872:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff9c1a78a783c0 x1874668973050112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828937.399879:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973050112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828937.399885:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828937.399886:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828937.399888:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973050112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828937.399891:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973050240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828937.399901:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828937.399906:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff9c1a78a7a1c0 x1874668973050240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828937.399913:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973050240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828937.399923:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828937.399925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828937.399927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973050240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828937.399931:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973050368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828937.399941:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828937.399946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff9c1a78a7b0c0 x1874668973050368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828937.399953:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973050368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828937.399959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828937.399961:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828937.399964:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973050368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828937.399991:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973050496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828937.400014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828937.400025:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828937/real 1787828937] req@ffff9c1a78a79e00 x1874668973050496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828937.400034:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973050496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828937.400040:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828937.400042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828937.400044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973050496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828937.911438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828937.911439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828938.167493:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828938.935424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828938.935437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828939.191536:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828939.959424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828939.959496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828940.215495:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828940.983427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828940.983429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828941.239509:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828942.007424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828942.007424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828942.263483:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828942.519443:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828942.519452:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828942.519456:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828942.519469:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5392 00000100:00080000:0.0:1787828942.519478:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828942.519485:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828942.519517:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973050624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828942.519541:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828942.519544:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828942.519546:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828942.519549:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5392 00000100:00080000:0.0:1787828942.519552:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828942.519554:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828942.519561:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828942.519573:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973050752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828942.519582:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828942.519585:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828942.519588:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828942.519589:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973050624:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828942.519594:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5392 00000100:00080000:0.0:1787828942.519604:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828942.519607:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828942.519611:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828942.519620:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973050880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828942.519628:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828942.519630:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828942.519632:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828942.519634:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5392 00000100:00080000:0.0:1787828942.519637:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828942.519639:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828942.519643:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828942.519652:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973051008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828942.519655:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828942.519664:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828942.519667:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828942.519668:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828942.519672:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5392 00000100:00080000:0.0:1787828942.519675:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828942.519677:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828942.519680:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828942.519688:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973051136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828942.519695:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828942.519698:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828942/real 1787828942] req@ffff9c1a78860b40 x1874668973050624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828942.519699:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828942.519702:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828942.519709:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5392 00000100:00080000:0.0:1787828942.519715:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828942.519718:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828942.519722:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828942.519724:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973050624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828942.519733:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973051264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828942.519735:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828942.519745:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828942.519748:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828942.519752:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973050624:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828942.519753:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828942.519757:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828942.519762:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5392 00000100:00080000:0.0:1787828942.519764:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828942.519767:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828942.519769:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828942.519776:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973051392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828942.519779:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973050752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828942.519812:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828942.519822:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828942/real 1787828942] req@ffff9c1a78862d00 x1874668973050752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828942.519831:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973050752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828942.519839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828942.519843:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828942.519847:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973050752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828942.519854:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973050880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828942.519877:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828942.519889:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828942/real 1787828942] req@ffff9c1a78860f00 x1874668973050880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828942.519902:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973050880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828942.519908:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828942.519911:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828942.519913:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973050880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828942.519917:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973051008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828942.519941:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828942.519949:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828942/real 1787828942] req@ffff9c1a78861a40 x1874668973051008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828942.519956:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973051008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828942.519963:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828942.519965:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828942.519967:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973051008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828942.519970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973051136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828942.519982:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828942.519988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828942/real 1787828942] req@ffff9c1a78862940 x1874668973051136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828942.519996:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973051136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828942.520002:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828942.520003:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828942.520005:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973051136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828942.520009:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973051264:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828942.520021:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828942.520027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828942/real 1787828942] req@ffff9c1a788612c0 x1874668973051264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828942.520035:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973051264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828942.520041:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828942.520042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828942.520044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973051264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828942.520060:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973051392:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828942.520070:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828942.520076:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828942/real 1787828942] req@ffff9c1a78863840 x1874668973051392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828942.520084:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973051392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828942.520094:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828942.520097:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828942.520099:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973051392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828943.031430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828943.031430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828943.287527:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828944.055415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828944.055415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828944.311482:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828945.079444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828945.079479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828945.335503:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828946.103485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828946.103505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828946.359505:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828947.127432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828947.127432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828947.383493:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828947.639481:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828947.639489:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828947.639493:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828947.639501:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5397 00000100:00080000:2.0:1787828947.639511:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828947.639517:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828947.639550:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973051520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828947.639576:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828947.639579:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828947.639580:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828947.639584:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5397 00000100:00080000:2.0:1787828947.639587:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828947.639590:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828947.639599:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828947.639612:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973051520:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828947.639612:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973051648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828947.639628:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828947.639630:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828947.639631:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828947.639634:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5397 00000100:00080000:2.0:1787828947.639636:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828947.639639:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828947.639642:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828947.639651:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973051776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828947.639659:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828947.639662:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828947.639663:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828947.639666:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5397 00000100:00080000:2.0:1787828947.639670:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828947.639674:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828947.639679:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828947.639680:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828947.639697:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973051904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828947.639709:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828947.639713:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828947.639715:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828947.639720:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5397 00000100:00080000:2.0:1787828947.639725:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828947.639730:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828947/real 1787828947] req@ffff9c1a49d8da40 x1874668973051520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828947.639734:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828947.639745:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828947.639753:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973051520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828947.639760:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973052032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828947.639799:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828947.639799:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828947.639806:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828947.639808:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787828947.639809:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828947.639816:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973051520:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828947.639817:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5397 00000100:00080000:2.0:1787828947.639825:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828947.639829:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828947.639833:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828947.639841:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973052160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828947.639847:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973051648:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828947.639856:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828947.639859:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828947.639860:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828947.639863:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5397 00000100:00080000:2.0:1787828947.639867:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828947.639870:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828947.639874:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828947.639878:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828947.639886:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973052288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828947.639892:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828947/real 1787828947] req@ffff9c1a49d8cb40 x1874668973051648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828947.639909:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973051648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828947.639917:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828947.639920:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828947.639924:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973051648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828947.639929:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973051776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828947.639941:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828947.639947:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828947/real 1787828947] req@ffff9c1a49d8cf00 x1874668973051776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828947.639955:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973051776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828947.639960:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828947.639962:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828947.639964:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973051776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828947.639969:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973051904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828947.639980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828947.639986:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828947/real 1787828947] req@ffff9c1a49d8c000 x1874668973051904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828947.639994:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973051904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828947.639999:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828947.640001:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828947.640002:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973051904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828947.640006:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973052032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787828947.640016:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828947.640022:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828947/real 1787828947] req@ffff9c1a49d8c780 x1874668973052032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828947.640029:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973052032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828947.640035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828947.640037:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828947.640038:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973052032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787828947.640049:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973052160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828947.640061:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828947.640067:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828947/real 1787828947] req@ffff9c1a49d8de00 x1874668973052160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828947.640075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973052160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828947.640081:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828947.640082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828947.640084:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973052160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787828947.640088:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973052288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787828947.640098:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828947.640103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828947/real 1787828947] req@ffff9c1a49d8da40 x1874668973052288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828947.640110:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973052288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828947.640116:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828947.640117:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828947.640119:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973052288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828948.151462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828948.151467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828948.407465:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828949.175426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828949.175452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828949.431485:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828950.199448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828950.199448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828950.455561:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828951.223434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828951.223434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828951.479504:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828952.247480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828952.247494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828952.503522:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828952.759551:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828952.759560:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828952.759564:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828952.759585:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5403 00000100:00080000:0.0:1787828952.759597:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828952.759604:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828952.759645:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973052416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828952.759676:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828952.759679:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828952.759680:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828952.759683:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5403 00000100:00080000:0.0:1787828952.759687:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828952.759690:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828952.759697:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828952.759707:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973052544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828952.759715:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828952.759717:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828952.759719:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828952.759723:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5403 00000100:00080000:0.0:1787828952.759724:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828952.759727:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828952.759730:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828952.759740:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973052672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828952.759747:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973052416:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828952.759752:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828952.759767:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828952.759771:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828952.759779:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5403 00000100:00080000:0.0:1787828952.759785:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828952.759792:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828952.759800:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828952.759815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973052800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828952.759825:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828952.759827:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828952.759829:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828952.759832:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5403 00000100:00080000:0.0:1787828952.759835:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828952.759838:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828952.759841:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828952.759849:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973052928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828952.759855:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828952.759857:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828952.759858:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828952.759861:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5403 00000100:00080000:0.0:1787828952.759863:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828952.759869:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828952.759876:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828952.759882:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828952.759890:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973053056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828952.759897:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828952.759900:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828952.759903:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828952.759906:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5403 00000100:00080000:0.0:1787828952.759908:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828952.759911:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828952.759936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828952/real 1787828952] req@ffff9c1a788621c0 x1874668973052416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828952.759938:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828952.759961:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973053184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828952.759970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973052416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.759984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.759991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828952.760000:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973052416:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787828952.760015:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973052544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828952.760044:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828952.760056:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828952/real 1787828952] req@ffff9c1a78860780 x1874668973052544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.760067:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973052544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.760076:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.760080:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828952.760087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973052544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828952.760118:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973052672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828952.760133:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828952.760181:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828952/real 1787828952] req@ffff9c1a78861e00 x1874668973052672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.760191:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973052672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.760198:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.760201:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828952.760203:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973052672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828952.760209:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973052800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828952.760226:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828952.760234:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828952/real 1787828952] req@ffff9c1a78862580 x1874668973052800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.760243:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973052800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.760250:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.760252:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828952.760254:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973052800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828952.760259:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973052928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828952.760272:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828952.760280:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828952/real 1787828952] req@ffff9c1a78861680 x1874668973052928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.760289:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973052928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.760295:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.760297:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828952.760300:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973052928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828952.760304:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973053056:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828952.760318:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828952.760325:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828952/real 1787828952] req@ffff9c1a788630c0 x1874668973053056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.760334:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973053056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.760340:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.760343:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828952.760345:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973053056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828952.760349:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973053184:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828952.760386:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828952.760399:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828952/real 1787828952] req@ffff9c1a78863c00 x1874668973053184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828952.760415:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973053184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828952.760426:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828952.760430:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828952.760435:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973053184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828953.271430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828953.271502:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828953.527585:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828954.295489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828954.295491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828954.551531:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828955.319444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828955.319447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828955.575560:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828956.343457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828956.343480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828956.599558:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828957.367439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828957.367507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828957.623529:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828957.879461:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828957.879473:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828957.879478:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828957.879492:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5408 00000100:00080000:0.0:1787828957.879506:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828957.879516:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828957.879558:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973053312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828957.879594:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828957.879600:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828957.879603:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828957.879610:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5408 00000100:00080000:0.0:1787828957.879615:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828957.879622:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828957.879634:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828957.879656:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973053440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828957.879673:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828957.879677:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973053312:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828957.879679:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828957.879686:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828957.879695:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5408 00000100:00080000:0.0:1787828957.879701:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828957.879707:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828957.879715:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828957.879734:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973053568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828957.879749:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828957.879754:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828957.879758:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828957.879760:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828957.879771:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5408 00000100:00080000:0.0:1787828957.879779:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828957.879785:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828957.879793:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828957.879810:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973053696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828957.879823:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828957.879826:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828957/real 1787828957] req@ffff9c1a78860000 x1874668973053312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828957.879832:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828957.879838:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828957.879851:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5408 00000100:00080000:0.0:1787828957.879861:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828957.879870:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828957.879877:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973053312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828957.879878:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828957.879894:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828957.879900:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787828957.879903:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973053824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828957.879906:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973053312:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787828957.879925:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828957.879929:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828957.879933:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828957.879937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973053440:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828957.879944:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5408 00000100:00080000:0.0:1787828957.879952:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828957.879958:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828957.879965:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828957.879971:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828957.879987:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973053952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828957.879991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828957/real 1787828957] req@ffff9c1a78863c00 x1874668973053440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828957.880031:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828957.880034:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973053440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828957.880042:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828957.880050:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828957.880051:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828957.880060:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828957.880065:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5408 00000100:00100000:1.0:1787828957.880068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973053440:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828957.880075:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828957.880079:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973053568:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828957.880086:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828957.880096:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828957.880120:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787828957.880138:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973054080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828957.880142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828957/real 1787828957] req@ffff9c1a788630c0 x1874668973053568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828957.880185:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973053568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828957.880206:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828957.880212:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828957.880215:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973053568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828957.880221:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973053696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828957.880243:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828957.880252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828957/real 1787828957] req@ffff9c1a78861680 x1874668973053696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828957.880262:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973053696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828957.880269:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828957.880271:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828957.880274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973053696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828957.880278:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973053824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828957.880290:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828957.880297:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828957/real 1787828957] req@ffff9c1a78862580 x1874668973053824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828957.880306:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973053824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828957.880312:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828957.880314:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828957.880318:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973053824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828957.880333:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973053952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828957.880379:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828957.880392:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828957/real 1787828957] req@ffff9c1a78861e00 x1874668973053952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828957.880402:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973053952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828957.880409:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828957.880412:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828957.880415:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973053952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828957.880422:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973054080:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828957.880442:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828957.880457:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828957/real 1787828957] req@ffff9c1a78860780 x1874668973054080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828957.880472:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973054080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828957.880483:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828957.880488:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828957.880492:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973054080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828958.391436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828958.391442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828958.647498:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828959.415425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828959.415425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828959.671523:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828960.439440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828960.439440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828960.695520:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828961.463449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828961.463450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828961.719496:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828962.487422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828962.487428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828962.743517:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828962.999481:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828962.999491:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828962.999496:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828962.999507:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5413 00000100:00080000:2.0:1787828962.999521:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828962.999530:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828962.999579:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973054208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828962.999611:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828962.999614:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828962.999616:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828962.999619:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5413 00000100:00080000:2.0:1787828962.999622:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828962.999626:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828962.999636:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828962.999645:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973054336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828962.999652:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828962.999655:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828962.999656:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828962.999659:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5413 00000100:00080000:2.0:1787828962.999661:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828962.999664:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828962.999667:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828962.999674:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973054464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828962.999683:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828962.999685:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828962.999686:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828962.999693:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973054208:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828962.999696:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5413 00000100:00080000:2.0:1787828962.999702:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828962.999705:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828962.999708:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828962.999716:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973054592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828962.999723:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828962.999726:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828962.999727:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828962.999730:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5413 00000100:00080000:2.0:1787828962.999733:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828962.999735:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828962.999738:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828962.999745:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973054720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828962.999751:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828962.999753:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828962.999754:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828962.999757:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5413 00000100:00080000:2.0:1787828962.999766:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828962.999768:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828962.999772:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828962.999781:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828962.999782:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973054848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828962.999791:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828962.999793:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828962.999795:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828962.999797:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5413 00000100:00080000:2.0:1787828962.999800:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828962.999803:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828962.999806:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828962.999815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973054976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828962.999841:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828962/real 1787828962] req@ffff9c1a49d8f0c0 x1874668973054208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828962.999859:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973054208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828962.999868:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828962.999872:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828962.999875:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973054208:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828962.999880:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973054336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828962.999899:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828962.999908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828962/real 1787828962] req@ffff9c1a49d8e1c0 x1874668973054336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828962.999916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973054336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828962.999922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828962.999924:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828962.999926:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973054336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828962.999949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973054464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828962.999961:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828962.999967:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828962/real 1787828962] req@ffff9c1a49d8c3c0 x1874668973054464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828962.999975:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973054464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828962.999980:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828962.999981:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828962.999983:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973054464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828962.999987:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973054592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828962.999997:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828963.000003:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828962/real 1787828962] req@ffff9c1a49d8e580 x1874668973054592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828963.000010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973054592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828963.000015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828963.000017:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828963.000019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973054592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828963.000022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973054720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828963.000032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828963.000038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828962/real 1787828963] req@ffff9c1a49d8fc00 x1874668973054720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828963.000046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973054720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828963.000051:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828963.000053:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828963.000054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973054720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828963.000058:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973054848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828963.000085:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828963.000092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828962/real 1787828963] req@ffff9c1a49d8ed00 x1874668973054848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828963.000107:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973054848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828963.000115:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828963.000117:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828963.000119:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973054848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828963.000123:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973054976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828963.000135:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828963.000141:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828962/real 1787828963] req@ffff9c1a49d8d680 x1874668973054976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828963.000149:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973054976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828963.000159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828963.000161:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828963.000163:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973054976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828963.511462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828963.511474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828963.767506:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828964.535425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828964.535425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828964.791490:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828965.559421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828965.559421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828965.815474:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828966.583443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828966.583443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828966.839494:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828967.607515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828967.607515:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828967.863547:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828968.119494:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828968.119504:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828968.119509:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828968.119519:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5418 00000100:00080000:3.0:1787828968.119531:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828968.119545:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828968.119583:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973055104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828968.119605:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828968.119608:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828968.119610:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828968.119615:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5418 00000100:00080000:3.0:1787828968.119619:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828968.119624:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828968.119639:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828968.119656:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973055232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828968.119666:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828968.119669:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828968.119671:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828968.119676:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5418 00000100:00080000:3.0:1787828968.119679:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828968.119682:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828968.119686:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828968.119695:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973055360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828968.119704:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828968.119709:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787828968.119710:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973055104:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828968.119711:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828968.119726:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5418 00000100:00080000:3.0:1787828968.119729:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828968.119732:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828968.119735:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828968.119746:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973055488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828968.119755:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828968.119783:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828968.119786:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828968.119792:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5418 00000100:00080000:3.0:1787828968.119798:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828968.119802:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828968.119807:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828968.119820:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973055616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828968.119830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828968.119833:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828968.119835:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828968.119840:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973055232:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828968.119840:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5418 00000100:00080000:3.0:1787828968.119848:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828968.119854:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828968.119859:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828968.119877:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973055744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828968.119884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828968.119887:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828968.119890:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828968.119895:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973055360:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828968.119898:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5418 00000100:00080000:3.0:1787828968.119911:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828968.119916:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828968.119921:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828968.119931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973055872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828968.119933:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973055488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828968.119972:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973055616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828968.119990:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973055744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828968.120007:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973055872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828968.631444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828968.631444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828968.887586:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828969.655422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828969.655424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828969.911514:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828970.679434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828970.679434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828970.935526:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828971.191591:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787828971.191670:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828971.191685:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828971.191719:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828971.191725:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828971.191730:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828971.191736:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828971.191743:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787828971.191750:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787828971.191775:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828971] req@ffff9c1a7895de00 x1874668973055872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828971.191797:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973055872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828971.191806:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828971.191810:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828971.191815:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973055872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828971.191824:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828971] req@ffff9c1a7895f0c0 x1874668973055360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828971.191854:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973055360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828971.191864:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828971.191867:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828971.191871:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973055360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828971.191878:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828971] req@ffff9c1a7895f840 x1874668973055488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828971.191892:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973055488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828971.191901:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828971.191905:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828971.191909:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973055488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828971.191927:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828971] req@ffff9c1a7895fc00 x1874668973055616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828971.191936:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973055616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828971.191942:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828971.191945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828971.191947:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973055616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828971.191951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828971] req@ffff9c1a7895e1c0 x1874668973055744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828971.191957:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973055744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828971.191962:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828971.191965:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828971.191968:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973055744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828971.191974:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828971] req@ffff9c1a7895da40 x1874668973055104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828971.191995:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973055104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828971.192004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828971.192025:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828971.192029:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973055104:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828971.192037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828968/real 1787828971] req@ffff9c1a7895cb40 x1874668973055232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828971.192050:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973055232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828971.192058:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828971.192062:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828971.192065:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973055232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828971.703547:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828971.703547:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828971.959607:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828972.727427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828972.727427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828972.983486:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828973.239423:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828973.239431:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828973.239434:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828973.239448:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5423 00000100:00080000:3.0:1787828973.239458:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828973.239465:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828973.239494:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973056000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828973.239517:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828973.239519:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828973.239521:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828973.239527:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5423 00000100:00080000:3.0:1787828973.239530:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828973.239532:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828973.239540:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828973.239549:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973056128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828973.239556:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828973.239559:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828973.239561:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828973.239564:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5423 00000100:00080000:3.0:1787828973.239566:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828973.239569:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828973.239572:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828973.239578:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973056000:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787828973.239581:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973056256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828973.239590:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828973.239593:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828973.239594:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828973.239596:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5423 00000100:00080000:3.0:1787828973.239598:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828973.239601:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828973.239607:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828973.239615:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973056384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828973.239624:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828973.239626:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828973.239628:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828973.239631:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5423 00000100:00080000:3.0:1787828973.239634:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828973.239636:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828973.239640:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828973.239646:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828973.239649:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973056512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828973.239657:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828973.239660:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828973.239661:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828973.239664:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5423 00000100:00080000:3.0:1787828973.239669:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828973.239673:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828973.239678:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828973.239696:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973056640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828973.239697:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828973] req@ffff9c1a7895c780 x1874668973056000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828973.239723:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828973.239729:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828973.239731:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828973.239734:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973056000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828973.239739:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5423 00000100:00080000:3.0:1787828973.239748:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828973.239750:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828973.239759:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828973.239759:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828973.239765:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973056000:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787828973.239770:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828973.239775:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973056128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787828973.239787:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973056768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828973.239809:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828973.239827:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828973] req@ffff9c1a7895e940 x1874668973056128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828973.239840:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973056128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828973.239849:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828973.239852:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828973.239855:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973056128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828973.239878:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973056256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828973.239897:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828973.239909:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828973] req@ffff9c1a7895c3c0 x1874668973056256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828973.239921:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973056256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828973.239928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828973.239931:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828973.239934:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973056256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828973.239941:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973056384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828973.239960:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828973.239972:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828973] req@ffff9c1a7895cf00 x1874668973056384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828973.239984:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973056384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828973.240012:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828973.240016:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828973.240020:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973056384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828973.240032:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973056512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828973.240053:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828973.240062:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828973] req@ffff9c1a7895f480 x1874668973056512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828973.240071:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973056512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828973.240079:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828973.240083:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828973.240087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973056512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828973.240110:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973056640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828973.240138:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828973.240151:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828973] req@ffff9c1a7895d2c0 x1874668973056640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828973.240165:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973056640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828973.240174:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828973.240176:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828973.240181:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973056640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828973.240188:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973056768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828973.240208:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828973.240220:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828973/real 1787828973] req@ffff9c1a7895c000 x1874668973056768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828973.240231:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973056768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828973.240237:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828973.240239:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828973.240241:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973056768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828973.751419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828973.751422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828974.007490:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828974.775426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828974.775428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828975.031563:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828975.799426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828975.799438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828976.055557:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828976.823434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828976.823439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828977.079554:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828977.847463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828977.847463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828978.103565:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828978.359465:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828978.359479:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828978.359484:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828978.359496:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5428 00000100:00080000:3.0:1787828978.359508:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828978.359514:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828978.359564:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973056896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828978.359599:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828978.359604:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828978.359606:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828978.359611:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5428 00000100:00080000:3.0:1787828978.359615:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828978.359620:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828978.359633:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828978.359646:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973057024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828978.359656:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973056896:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828978.359656:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828978.359662:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828978.359666:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828978.359670:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5428 00000100:00080000:3.0:1787828978.359673:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828978.359675:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828978.359678:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828978.359687:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973057152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828978.359695:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828978.359698:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828978.359700:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828978.359702:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5428 00000100:00080000:3.0:1787828978.359706:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828978.359708:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828978.359711:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828978.359718:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973057280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828978.359725:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828978.359727:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828978.359728:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828978.359733:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828978.359733:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5428 00000100:00080000:3.0:1787828978.359738:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828978.359741:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828978.359745:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828978.359753:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973057408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828978.359759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828978.359762:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828978.359763:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828978.359766:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5428 00000100:00080000:3.0:1787828978.359768:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828978.359770:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828978.359775:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828978.359786:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973057536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828978.359788:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff9c1a7895e580 x1874668973056896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828978.359796:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828978.359802:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828978.359805:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828978.359812:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5428 00000100:00080000:3.0:1787828978.359816:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828978.359818:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828978.359822:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828978.359829:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973056896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828978.359830:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973057664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828978.359865:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828978.359871:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828978.359874:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973056896:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828978.359898:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973057024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828978.359917:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828978.359926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff9c1a7895c000 x1874668973057024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828978.359935:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973057024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828978.359941:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828978.359943:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828978.359945:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973057024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828978.359949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973057152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828978.359977:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828978.359986:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff9c1a7895d2c0 x1874668973057152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828978.359994:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973057152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828978.360000:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828978.360002:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828978.360004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973057152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828978.360008:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973057280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828978.360021:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828978.360027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff9c1a7895f480 x1874668973057280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828978.360035:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973057280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828978.360041:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828978.360042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828978.360044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973057280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828978.360048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973057408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828978.360058:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828978.360064:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff9c1a7895cf00 x1874668973057408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828978.360071:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973057408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828978.360077:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828978.360078:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828978.360080:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973057408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828978.360084:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973057536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828978.360094:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828978.360100:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff9c1a7895c3c0 x1874668973057536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828978.360113:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973057536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828978.360118:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828978.360119:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828978.360121:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973057536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828978.360125:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973057664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828978.360137:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828978.360148:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828978/real 1787828978] req@ffff9c1a7895e940 x1874668973057664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828978.360160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973057664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828978.360169:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828978.360173:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828978.360177:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973057664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828978.871442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828978.871442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828979.127510:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828979.895454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828979.895473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828980.151558:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828980.919443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828980.919476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828981.175488:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828981.943459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828981.943459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828982.199542:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828982.967459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828982.967459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828983.223494:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828983.479459:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828983.479470:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828983.479474:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828983.479486:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5433 00000100:00080000:0.0:1787828983.479497:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828983.479504:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787828983.479549:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973057792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828983.479579:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828983.479584:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828983.479587:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828983.479592:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5433 00000100:00080000:0.0:1787828983.479596:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828983.479602:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828983.479615:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828983.479630:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973057792:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828983.479634:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973057920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828983.479649:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828983.479655:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828983.479657:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828983.479661:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5433 00000100:00080000:0.0:1787828983.479666:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828983.479672:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828983.479677:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828983.479694:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973058048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828983.479707:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828983.479711:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828983.479713:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828983.479717:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5433 00000100:00080000:0.0:1787828983.479723:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828983.479723:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828983.479732:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828983.479739:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828983.479753:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973058176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828983.479765:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828983.479769:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828983.479771:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828983.479773:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828983/real 1787828983] req@ffff9c1a788621c0 x1874668973057792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828983.479781:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5433 00000100:00080000:0.0:1787828983.479789:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828983.479797:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973057792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828983.479798:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787828983.479810:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828983.479817:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787828983.479820:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828983.479825:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973057792:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787828983.479839:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973058304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828983.479853:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973057920:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828983.479854:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828983.479863:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828983.479866:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828983.479872:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5433 00000100:00080000:0.0:1787828983.479877:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828983.479881:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787828983.479887:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828983.479895:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828983.479896:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828983/real 1787828983] req@ffff9c1a78863840 x1874668973057920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828983.479910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973057920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828983.479915:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973058432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828983.479947:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828983.479950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828983.479954:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787828983.479956:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828983.479959:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828983.479960:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973057920:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828983.479966:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5433 00000100:00080000:0.0:1787828983.479971:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828983.479971:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973058048:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787828983.479979:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787828983.479985:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828983.479997:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973058560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828983.480003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828983.480018:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828983/real 1787828983] req@ffff9c1a788612c0 x1874668973058048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828983.480032:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973058048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828983.480037:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828983.480039:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828983.480041:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973058048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828983.480045:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973058176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828983.480057:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828983.480064:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828983/real 1787828983] req@ffff9c1a78862940 x1874668973058176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828983.480072:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973058176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828983.480085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828983.480087:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828983.480089:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973058176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828983.480101:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973058304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828983.480113:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828983.480120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828983/real 1787828983] req@ffff9c1a78861a40 x1874668973058304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828983.480127:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973058304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828983.480138:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828983.480141:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828983.480145:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973058304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828983.480152:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973058432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828983.480172:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828983.480184:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828983/real 1787828983] req@ffff9c1a78860f00 x1874668973058432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828983.480198:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973058432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828983.480204:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828983.480206:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828983.480209:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973058432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828983.480213:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973058560:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828983.480228:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828983.480245:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828983/real 1787828983] req@ffff9c1a78862d00 x1874668973058560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828983.480253:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973058560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828983.480258:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828983.480260:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828983.480263:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973058560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828983.991426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828983.991431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828984.247520:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828985.015477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828985.015477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828985.271532:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828986.039417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828986.039418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828986.295509:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828987.063440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828987.063440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828987.319526:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828988.087420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787828988.087435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828988.343481:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828988.599506:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828988.599514:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828988.599517:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828988.599526:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5438 00000100:00080000:2.0:1787828988.599535:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828988.599541:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787828988.599575:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973058688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828988.599599:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828988.599602:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828988.599603:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828988.599607:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5438 00000100:00080000:2.0:1787828988.599609:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828988.599612:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828988.599619:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828988.599628:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973058816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828988.599636:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828988.599639:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828988.599641:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828988.599649:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5438 00000100:00100000:1.0:1787828988.599652:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973058688:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828988.599653:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828988.599657:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828988.599664:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828988.599672:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973058944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828988.599684:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828988.599688:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828988.599690:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828988.599694:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5438 00000100:00080000:2.0:1787828988.599699:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828988.599704:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828988.599709:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828988.599722:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973059072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828988.599732:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787828988.599740:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828988.599745:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828988.599747:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828988.599753:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5438 00000100:00080000:2.0:1787828988.599758:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828988.599763:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828988.599768:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828988.599784:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828988/real 1787828988] req@ffff9c1a49d8d680 x1874668973058688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828988.599784:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973059200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828988.599806:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828988.599810:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828988.599812:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828988.599816:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5438 00000100:00080000:2.0:1787828988.599819:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828988.599820:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973058688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828988.599825:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828988.599833:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828988.599838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828988.599843:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973059328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828988.599845:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828988.599854:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973058688:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787828988.599857:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828988.599862:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828988.599863:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828988.599865:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973058816:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787828988.599869:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5438 00000100:00080000:2.0:1787828988.599873:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828988.599875:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787828988.599879:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828988.599904:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973059456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828988.599916:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828988.599938:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828988/real 1787828988] req@ffff9c1a49d8ed00 x1874668973058816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828988.599964:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973058816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828988.599980:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828988.599984:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828988.599988:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973058816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828988.600010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973058944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828988.600027:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828988.600035:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828988/real 1787828988] req@ffff9c1a49d8fc00 x1874668973058944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828988.600043:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973058944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828988.600048:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828988.600050:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828988.600052:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973058944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828988.600056:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973059072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828988.600066:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828988.600072:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828988/real 1787828988] req@ffff9c1a49d8e580 x1874668973059072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828988.600080:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973059072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828988.600085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828988.600086:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828988.600089:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973059072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828988.600092:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973059200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787828988.600102:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828988.600108:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828988/real 1787828988] req@ffff9c1a49d8c3c0 x1874668973059200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828988.600115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973059200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828988.600120:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828988.600121:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828988.600123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973059200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787828988.600126:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973059328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828988.600137:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828988.600152:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828988/real 1787828988] req@ffff9c1a49d8e1c0 x1874668973059328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828988.600159:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973059328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828988.600164:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828988.600166:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828988.600167:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973059328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787828988.600171:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973059456:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787828988.600181:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828988.600192:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828988/real 1787828988] req@ffff9c1a49d8f0c0 x1874668973059456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828988.600204:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973059456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828988.600213:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828988.600215:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787828988.600219:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973059456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828989.111450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828989.111485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828989.367522:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828990.135459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828990.135467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828990.391504:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828991.159457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828991.159457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828991.415590:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828992.183473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828992.183539:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828992.439525:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828993.207500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828993.207500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828993.463536:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828993.719467:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828993.719478:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828993.719485:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828993.719499:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5444 00000100:00080000:1.0:1787828993.719513:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828993.719523:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787828993.719574:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973059584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828993.719598:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828993.719602:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828993.719604:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828993.719608:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5444 00000100:00080000:1.0:1787828993.719612:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828993.719616:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828993.719628:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828993.719639:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973059584:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828993.719648:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973059712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828993.719658:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828993.719661:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828993.719662:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828993.719666:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5444 00000100:00080000:1.0:1787828993.719669:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828993.719672:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828993.719676:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828993.719686:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973059840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828993.719696:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828993.719699:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828993.719701:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828993.719705:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5444 00000100:00080000:1.0:1787828993.719707:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828993.719710:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828993.719714:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828993.719723:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787828993.719725:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973059968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828993.719734:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828993.719737:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828993.719739:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828993.719744:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5444 00000100:00080000:1.0:1787828993.719747:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828993.719751:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828993.719756:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828993.719765:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973060096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828993.719771:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828993.719776:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828993.719776:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828993/real 1787828993] req@ffff9c1a78a79e00 x1874668973059584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828993.719781:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828993.719790:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5444 00000100:00080000:1.0:1787828993.719794:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828993.719797:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828993.719802:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828993.719803:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973059584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828993.719815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973060224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828993.719816:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828993.719825:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787828993.719830:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828993.719833:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828993.719835:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828993.719838:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5444 00000100:00080000:1.0:1787828993.719840:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828993.719842:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787828993.719845:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828993.719862:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973059584:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787828993.719868:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973060352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828993.719893:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973059712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828993.719917:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828993.719931:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828993/real 1787828993] req@ffff9c1a78a7b0c0 x1874668973059712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828993.719940:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973059712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828993.719947:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828993.719950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828993.719953:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973059712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828993.719959:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973059840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828993.719973:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828993.719984:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828993/real 1787828993] req@ffff9c1a78a7a1c0 x1874668973059840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828993.719996:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973059840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828993.720004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828993.720007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828993.720010:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973059840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828993.720016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973059968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828993.720036:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828993.720048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828993/real 1787828993] req@ffff9c1a78a783c0 x1874668973059968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828993.720059:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973059968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828993.720069:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828993.720072:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828993.720075:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973059968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828993.720081:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973060096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828993.720102:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828993.720115:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828993/real 1787828993] req@ffff9c1a78a7a940 x1874668973060096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828993.720127:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973060096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828993.720136:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828993.720139:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828993.720142:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973060096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828993.720149:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973060224:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828993.720172:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828993.720186:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828993/real 1787828993] req@ffff9c1a78a78780 x1874668973060224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828993.720200:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973060224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828993.720226:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828993.720230:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828993.720234:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973060224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828993.720243:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973060352:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828993.720268:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828993.720283:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828993/real 1787828993] req@ffff9c1a78a7bc00 x1874668973060352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828993.720295:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973060352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828993.720304:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828993.720307:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828993.720310:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973060352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828994.231456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828994.231456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828994.487545:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828995.255422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828995.255439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787828995.511471:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828996.279419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787828996.279424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828996.535514:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828997.303490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828997.303504:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787828997.559497:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828998.327416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787828998.327426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787828998.583514:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828998.839479:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828998.839488:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828998.839492:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828998.839500:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5449 00000100:00080000:3.0:1787828998.839509:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828998.839516:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787828998.839556:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973060480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828998.839588:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828998.839592:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828998.839594:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828998.839599:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5449 00000100:00080000:3.0:1787828998.839603:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828998.839608:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828998.839617:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828998.839629:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973060608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828998.839638:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828998.839639:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973060480:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828998.839641:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828998.839646:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828998.839653:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5449 00000100:00080000:3.0:1787828998.839657:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828998.839661:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828998.839667:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828998.839680:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973060736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828998.839691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828998.839694:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828998.839696:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828998.839700:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5449 00000100:00080000:3.0:1787828998.839705:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828998.839712:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787828998.839712:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828998.839721:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828998.839734:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973060864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828998.839742:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828998.839745:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828998.839746:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828998.839750:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828998/real 1787828998] req@ffff9c1a7895c780 x1874668973060480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828998.839753:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5449 00000100:00080000:3.0:1787828998.839758:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828998.839764:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828998.839770:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828998.839776:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973060480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828998.839782:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973060992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828998.839788:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828998.839791:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828998.839794:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787828998.839796:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828998.839799:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828998.839800:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973060480:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787828998.839806:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5449 00000100:00080000:3.0:1787828998.839809:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828998.839812:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828998.839827:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828998.839836:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973061120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828998.839844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828998.839845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973060608:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787828998.839852:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828998.839855:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828998.839859:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5449 00000100:00080000:3.0:1787828998.839862:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828998.839865:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787828998.839868:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828998.839874:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787828998.839879:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973061248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828998.839888:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828998/real 1787828998] req@ffff9c1a7895cb40 x1874668973060608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828998.839904:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973060608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828998.839910:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828998.839912:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828998.839914:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973060608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828998.839919:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973060736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828998.839931:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828998.839938:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828998/real 1787828998] req@ffff9c1a7895da40 x1874668973060736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828998.839945:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973060736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828998.839950:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828998.839952:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828998.839954:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973060736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828998.839958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973060864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828998.839968:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828998.839973:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828998/real 1787828998] req@ffff9c1a7895e1c0 x1874668973060864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828998.839981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973060864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828998.839991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828998.839993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828998.839995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973060864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828998.839998:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973060992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787828998.840008:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828998.840013:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828998/real 1787828998] req@ffff9c1a7895fc00 x1874668973060992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828998.840024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973060992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828998.840032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828998.840036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828998.840040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973060992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787828998.840060:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973061120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828998.840081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828998.840089:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828998/real 1787828998] req@ffff9c1a7895f840 x1874668973061120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828998.840097:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973061120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828998.840103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828998.840105:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828998.840108:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973061120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828998.840112:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973061248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787828998.840122:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787828998.840131:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828998/real 1787828998] req@ffff9c1a7895f0c0 x1874668973061248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828998.840143:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973061248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828998.840153:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828998.840156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787828998.840160:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973061248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787828999.351484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787828999.351507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787828999.607527:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829000.375428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829000.375457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829000.631495:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829001.399460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829001.399516:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829001.655568:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829002.423466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829002.423466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829002.679448:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829003.447439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829003.447440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829003.703531:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829003.959482:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829003.959490:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829003.959494:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829003.959503:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5454 00000100:00080000:0.0:1787829003.959512:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829003.959518:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829003.959550:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973061376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829003.959576:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829003.959578:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829003.959580:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829003.959583:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5454 00000100:00080000:0.0:1787829003.959585:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829003.959588:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829003.959595:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829003.959605:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973061504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829003.959612:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829003.959616:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829003.959617:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829003.959620:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5454 00000100:00080000:0.0:1787829003.959622:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829003.959626:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973061376:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829003.959627:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829003.959634:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829003.959642:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973061632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829003.959650:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829003.959653:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829003.959654:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829003.959657:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5454 00000100:00080000:0.0:1787829003.959659:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829003.959661:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829003.959664:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829003.959672:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973061760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829003.959678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829003.959680:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829003.959681:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829003.959684:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5454 00000100:00080000:0.0:1787829003.959686:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829003.959690:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829003.959694:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829003.959699:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829003.959707:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973061888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829003.959713:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829003.959715:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829003.959716:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829003.959719:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5454 00000100:00080000:0.0:1787829003.959722:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829003.959724:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829003.959728:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829003.959730:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829003/real 1787829003] req@ffff9c1a78860b40 x1874668973061376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829003.959738:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973062016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829003.959747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829003.959752:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829003.959752:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973061376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829003.959754:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829003.959762:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5454 00000100:00080000:0.0:1787829003.959768:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829003.959772:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829003.959772:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829003.959778:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829003.959795:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829003.959800:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973061376:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829003.959806:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973062144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829003.959809:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973061504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829003.959836:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829003.959849:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829003/real 1787829003] req@ffff9c1a78862d00 x1874668973061504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829003.959858:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973061504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829003.959864:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829003.959866:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829003.959869:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973061504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829003.959888:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973061632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829003.959899:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829003.959905:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829003/real 1787829003] req@ffff9c1a78860f00 x1874668973061632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829003.959912:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973061632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829003.959918:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829003.959920:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829003.959922:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973061632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829003.959926:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973061760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829003.959936:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829003.959942:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829003/real 1787829003] req@ffff9c1a78861a40 x1874668973061760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829003.959950:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973061760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829003.959955:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829003.959957:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829003.959958:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973061760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829003.959962:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973061888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829003.959972:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829003.959978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829003/real 1787829003] req@ffff9c1a78862940 x1874668973061888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829003.959985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973061888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829003.959990:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829003.959992:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829003.959994:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973061888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829003.959997:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973062016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829003.960008:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829003.960014:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829003/real 1787829003] req@ffff9c1a788612c0 x1874668973062016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829003.960026:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973062016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829003.960040:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829003.960042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829003.960044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973062016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829003.960047:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973062144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829003.960059:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829003.960065:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829003/real 1787829003] req@ffff9c1a78863840 x1874668973062144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829003.960073:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973062144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829003.960078:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829003.960081:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829003.960083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973062144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829004.471451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829004.471496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829004.727530:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829005.495431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829005.495447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829005.751532:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829006.519462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829006.519462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787829006.775546:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829007.543431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829007.543462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829007.799519:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829008.567448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829008.567448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829008.823511:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829009.079435:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829009.079445:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829009.079450:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829009.079461:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5459 00000100:00080000:0.0:1787829009.079472:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829009.079480:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829009.079527:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973062272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829009.079561:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829009.079565:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829009.079567:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829009.079573:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5459 00000100:00080000:0.0:1787829009.079577:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829009.079582:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829009.079595:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829009.079610:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973062400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829009.079620:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829009.079624:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829009.079627:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829009.079632:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5459 00000100:00080000:0.0:1787829009.079636:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829009.079643:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829009.079649:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973062272:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829009.079652:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829009.079680:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973062528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829009.079699:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829009.079703:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829009.079706:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829009.079712:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5459 00000100:00080000:0.0:1787829009.079718:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829009.079723:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829009.079730:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829009.079734:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829009.079775:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973062656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829009.079792:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829009.079796:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829009.079799:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829009.079803:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff9c1a788621c0 x1874668973062272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829009.079810:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5459 00000100:00080000:0.0:1787829009.079824:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829009.079832:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829009.079836:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973062272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829009.079846:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829009.079851:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829009.079857:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829009.079861:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973062272:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829009.079868:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973062784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829009.079871:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973062400:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829009.079895:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829009.079899:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829009.079907:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829009.079908:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829009.079921:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5459 00000100:00100000:2.0:1787829009.079925:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff9c1a78860780 x1874668973062400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829009.079932:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829009.079951:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829009.079951:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973062400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829009.079973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829009.079974:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829009.079984:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829009.079988:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973062400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829009.079999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973062912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829009.080014:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973062528:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829009.080015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829009.080022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829009.080026:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829009.080034:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5459 00000800:00000100:2.0:1787829009.080035:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829009.080048:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829009.080054:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff9c1a78861e00 x1874668973062528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829009.080060:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829009.080081:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829009.080086:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973062528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829009.080100:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829009.080102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973063040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829009.080106:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829009.080118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973062528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829009.080140:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973062656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829009.080164:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829009.080177:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff9c1a78862580 x1874668973062656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829009.080185:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973062656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829009.080191:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829009.080193:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829009.080195:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973062656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829009.080199:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973062784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829009.080210:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829009.080216:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff9c1a78861680 x1874668973062784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829009.080224:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973062784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829009.080229:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829009.080231:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829009.080233:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973062784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829009.080244:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973062912:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829009.080256:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829009.080262:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff9c1a788630c0 x1874668973062912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829009.080269:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973062912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829009.080274:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829009.080276:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829009.080278:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973062912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829009.080282:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973063040:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829009.080291:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829009.080297:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829009/real 1787829009] req@ffff9c1a78863c00 x1874668973063040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829009.080305:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973063040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829009.080310:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829009.080312:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829009.080314:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973063040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829009.591443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829009.591472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829009.847489:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829010.615451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829010.615478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829010.871542:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829011.639468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829011.639469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829011.895586:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829012.663476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829012.663485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829012.919603:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829013.687508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829013.687519:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787829013.943620:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829014.199541:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829014.199561:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829014.199569:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829014.199588:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5464 00000100:00080000:0.0:1787829014.199604:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829014.199614:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829014.199686:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973063168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829014.199766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829014.199775:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829014.199779:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829014.199788:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5464 00000100:00080000:0.0:1787829014.199798:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829014.199807:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829014.199835:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973063168:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829014.199841:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829014.199875:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973063296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829014.199891:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829014.199896:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829014.199899:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829014.199909:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829014.199925:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5464 00000100:00080000:0.0:1787829014.199935:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829014.199944:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829014.199953:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff9c1a78860000 x1874668973063168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829014.199957:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829014.199982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973063168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829014.200005:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829014.200013:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829014.200014:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973063424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829014.200029:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973063168:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829014.200034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829014.200040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829014.200043:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829014.200048:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5464 00000100:00100000:2.0:1787829014.200061:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973063296:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829014.200071:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829014.200084:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829014.200096:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787829014.200101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829014.200111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff9c1a78863c00 x1874668973063296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829014.200122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973063552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829014.200130:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973063296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829014.200183:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829014.200190:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829014.200193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829014.200195:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973063296:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829014.200211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829014.200217:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829014.200222:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973063424:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829014.200229:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5464 00000100:00080000:0.0:1787829014.200240:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829014.200248:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829014.200255:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829014.200267:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829014.200273:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973063680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829014.200286:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829014.200288:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff9c1a788630c0 x1874668973063424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829014.200295:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829014.200303:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829014.200316:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5464 00000100:00080000:0.0:1787829014.200325:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829014.200335:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829014.200341:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973063424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829014.200381:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829014.200382:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829014.200396:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829014.200404:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973063424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829014.200417:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973063808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829014.200429:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973063552:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829014.200458:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829014.200468:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787829014.200469:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829014.200480:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829014.200487:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff9c1a78861680 x1874668973063552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829014.200493:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5464 00000100:00080000:0.0:1787829014.200513:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829014.200519:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973063552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829014.200530:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829014.200534:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829014.200540:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787829014.200544:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829014.200545:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973063552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829014.200564:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973063680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829014.200583:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973063936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829014.200587:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829014.200606:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff9c1a78862580 x1874668973063680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829014.200625:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973063680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829014.200633:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829014.200636:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829014.200638:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973063680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829014.200651:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973063808:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829014.200663:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829014.200670:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff9c1a78861e00 x1874668973063808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829014.200678:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973063808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829014.200683:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829014.200685:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829014.200687:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973063808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829014.200691:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973063936:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829014.200701:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829014.200707:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829014/real 1787829014] req@ffff9c1a78860780 x1874668973063936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829014.200731:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973063936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829014.200742:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829014.200744:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829014.200746:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973063936:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829014.711501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829014.711504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829014.967551:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829015.735447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829015.735452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829015.991593:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829016.759515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829016.759515:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829017.015497:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829017.783417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829017.783417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829018.039617:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829018.807457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829018.807498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829019.063578:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829019.319507:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.319518:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829019.319523:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829019.319535:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5469 00000100:00080000:1.0:1787829019.319549:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.319558:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829019.319598:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973064064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829019.319630:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.319634:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829019.319635:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829019.319639:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5469 00000100:00080000:1.0:1787829019.319643:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.319647:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829019.319656:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829019.319667:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973064192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829019.319694:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.319698:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829019.319698:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973064064:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829019.319702:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829019.319710:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5469 00000100:00080000:1.0:1787829019.319714:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.319717:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829019.319724:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829019.319737:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973064320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829019.319747:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.319750:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829019.319752:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829019.319754:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5469 00000100:00080000:1.0:1787829019.319758:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.319760:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829019.319764:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829019.319770:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829019.319775:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973064448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829019.319787:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.319790:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829019.319792:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829019.319796:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5469 00000100:00080000:1.0:1787829019.319799:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.319802:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829019.319805:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829019.319814:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973064576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829019.319821:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829019.319824:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829019.319829:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829019.319830:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff9c1a78a7b840 x1874668973064064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829019.319835:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5469 00000100:00080000:1.0:1787829019.319841:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.319849:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829019.319856:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829019.319863:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973064064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829019.319868:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973064704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829019.319896:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829019.319896:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829019.319904:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829019.319908:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829019.319908:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787829019.319920:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5469 00000100:00100000:2.0:1787829019.319920:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973064064:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829019.319926:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829019.319932:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829019.319936:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829019.319945:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973064832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829019.319955:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973064192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829019.319980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829019.319997:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff9c1a78a792c0 x1874668973064192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829019.320025:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973064192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829019.320037:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829019.320042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829019.320046:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973064192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829019.320055:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973064320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829019.320075:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829019.320085:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff9c1a78a79a40 x1874668973064320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829019.320096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973064320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829019.320103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829019.320106:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829019.320108:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973064320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829019.320113:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973064448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829019.320126:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829019.320132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff9c1a78a7b480 x1874668973064448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829019.320142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973064448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829019.320155:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829019.320157:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829019.320160:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973064448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829019.320164:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973064576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829019.320175:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829019.320182:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff9c1a78a78f00 x1874668973064576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829019.320191:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973064576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829019.320201:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829019.320205:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829019.320210:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973064576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829019.320217:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973064704:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829019.320243:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829019.320257:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff9c1a78a7a580 x1874668973064704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829019.320269:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973064704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829019.320276:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829019.320278:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829019.320283:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973064704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829019.320310:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973064832:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829019.320337:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829019.320386:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829019/real 1787829019] req@ffff9c1a78a78000 x1874668973064832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829019.320397:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973064832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829019.320405:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829019.320407:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829019.320410:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973064832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829019.831517:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829019.831519:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829020.087534:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829020.855505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829020.855507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787829021.111574:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829021.879487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829021.879488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829022.135544:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829022.903438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829022.903471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829023.159517:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829023.927495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829023.927499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829024.183554:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829024.439488:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829024.439501:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829024.439506:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829024.439518:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5474 00000100:00080000:0.0:1787829024.439532:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829024.439541:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829024.439604:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973064960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829024.439652:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829024.439656:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829024.439658:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829024.439663:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5474 00000100:00080000:0.0:1787829024.439667:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829024.439672:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829024.439683:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829024.439699:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973065088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829024.439712:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829024.439715:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829024.439717:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829024.439725:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5474 00000100:00100000:2.0:1787829024.439728:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973064960:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829024.439729:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829024.439756:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829024.439761:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829024.439774:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973065216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829024.439785:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829024.439788:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829024.439790:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829024.439793:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5474 00000100:00080000:0.0:1787829024.439796:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829024.439800:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829024.439805:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829024.439814:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973065344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829024.439822:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829024.439826:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829024.439827:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829024.439830:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5474 00000100:00080000:0.0:1787829024.439834:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829024.439838:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829024.439846:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787829024.439847:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829024.439877:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973065472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829024.439901:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829024.439906:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829024.439909:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829024.439916:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5474 00000100:00080000:0.0:1787829024.439924:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829024.439924:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff9c1a788621c0 x1874668973064960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829024.439938:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829024.439949:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829024.439952:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973064960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829024.439965:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829024.439969:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973065600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829024.439974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829024.439984:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973064960:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829024.439993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829024.439998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829024.440000:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829024.440005:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5474 00000100:00080000:0.0:1787829024.440009:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829024.440013:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829024.440018:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829024.440021:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973065088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829024.440032:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973065728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829024.440053:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829024.440068:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff9c1a78863840 x1874668973065088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829024.440081:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973065088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829024.440089:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829024.440093:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829024.440096:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973065088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829024.440101:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973065216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829024.440115:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829024.440123:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff9c1a788612c0 x1874668973065216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829024.440133:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973065216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829024.440139:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829024.440141:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829024.440144:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973065216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829024.440149:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973065344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829024.440162:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829024.440170:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff9c1a78862940 x1874668973065344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829024.440179:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973065344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829024.440186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829024.440188:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829024.440191:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973065344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829024.440195:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973065472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829024.440208:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829024.440215:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff9c1a78861a40 x1874668973065472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829024.440225:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973065472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829024.440231:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829024.440234:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829024.440236:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973065472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829024.440240:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973065600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829024.440254:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829024.440261:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff9c1a78860f00 x1874668973065600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829024.440277:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973065600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829024.440283:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829024.440285:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829024.440288:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973065600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829024.440305:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973065728:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829024.440318:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829024.440325:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829024/real 1787829024] req@ffff9c1a78862d00 x1874668973065728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829024.440335:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973065728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829024.440342:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829024.440382:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829024.440386:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973065728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829024.951470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829024.951471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829025.207574:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829025.975469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829025.975469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829026.231623:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829026.999432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829026.999451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829027.255501:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829028.023459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829028.023459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787829028.279537:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829029.047448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829029.047448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829029.303503:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829029.559405:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829029.559414:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829029.559420:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829029.559432:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5479 00000100:00080000:0.0:1787829029.559444:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829029.559451:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829029.559493:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973065856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829029.559531:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829029.559535:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829029.559536:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829029.559540:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5479 00000100:00080000:0.0:1787829029.559543:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829029.559546:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829029.559557:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829029.559570:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973065984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829029.559579:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829029.559581:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973065856:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829029.559582:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829029.559586:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829029.559592:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5479 00000100:00080000:0.0:1787829029.559594:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829029.559597:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829029.559611:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829029.559623:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973066112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829029.559642:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829029.559645:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829029.559646:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829029.559649:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5479 00000100:00080000:0.0:1787829029.559653:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829029.559657:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829029.559661:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829029.559671:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973066240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829029.559678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829029.559681:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829029.559682:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829029.559685:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5479 00000100:00080000:0.0:1787829029.559688:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829029.559703:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829029.559704:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787829029.559707:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829029.559717:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973066368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829029.559725:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829029.559728:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829029.559730:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829029.559734:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5479 00000100:00080000:0.0:1787829029.559737:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829029.559740:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829029.559744:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829029.559753:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973066496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829029.559760:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829029.559763:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829029.559764:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829029.559768:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5479 00000100:00080000:0.0:1787829029.559771:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829029.559774:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829029.559780:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829029.559780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829029/real 1787829029] req@ffff9c1a78860b40 x1874668973065856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829029.559790:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973066624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829029.559815:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973065856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829029.559828:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829029.559834:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829029.559839:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973065856:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829029.559870:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973065984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829029.559894:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829029.559917:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829029/real 1787829029] req@ffff9c1a78862d00 x1874668973065984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829029.559926:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973065984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829029.559934:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829029.559937:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829029.559940:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973065984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829029.559946:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973066112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829029.559962:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829029.559976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829029/real 1787829029] req@ffff9c1a78860f00 x1874668973066112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829029.559988:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973066112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829029.560001:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829029.560005:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829029.560008:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973066112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829029.560015:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973066240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829029.560046:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829029.560056:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829029/real 1787829029] req@ffff9c1a78861a40 x1874668973066240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829029.560074:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973066240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829029.560080:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829029.560082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829029.560084:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973066240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829029.560088:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973066368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829029.560101:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829029.560107:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829029/real 1787829029] req@ffff9c1a78862940 x1874668973066368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829029.560114:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973066368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829029.560121:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829029.560123:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829029.560126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973066368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829029.560129:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973066496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829029.560143:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829029.560155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829029/real 1787829029] req@ffff9c1a788612c0 x1874668973066496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829029.560167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973066496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829029.560177:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829029.560181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829029.560185:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973066496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829029.560192:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973066624:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829029.560216:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829029.560230:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829029/real 1787829029] req@ffff9c1a78863840 x1874668973066624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829029.560248:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973066624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829029.560257:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829029.560259:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829029.560262:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973066624:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829030.071485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829030.071485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829030.327570:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829031.095442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829031.095442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829031.351486:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829032.119470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829032.119470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829032.375507:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829033.143453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829033.143453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829033.399644:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829034.167448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829034.167448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829034.423548:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829034.679474:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829034.679489:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829034.679494:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829034.679506:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5485 00000100:00080000:1.0:1787829034.679517:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829034.679524:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829034.679586:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973066752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829034.679613:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829034.679617:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829034.679623:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829034.679629:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5485 00000100:00080000:1.0:1787829034.679633:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829034.679638:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829034.679651:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829034.679665:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973066880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829034.679668:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973066752:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829034.679675:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829034.679679:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829034.679680:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829034.679684:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5485 00000100:00080000:1.0:1787829034.679686:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829034.679690:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829034.679694:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829034.679702:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973067008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829034.679712:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829034.679715:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829034.679716:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829034.679720:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5485 00000100:00080000:1.0:1787829034.679722:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829034.679726:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829034.679730:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829034.679738:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973067136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829034.679746:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829034.679749:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829034.679751:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829034.679755:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5485 00000100:00080000:1.0:1787829034.679758:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829034.679762:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829034.679767:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829034.679777:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973067264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829034.679786:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829034.679790:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829034.679790:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973066880:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829034.679792:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829034.679796:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5485 00000100:00080000:1.0:1787829034.679800:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829034.679805:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829034.679811:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829034.679820:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973067008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829034.679821:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973067392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829034.679831:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829034.679835:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829034.679837:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829034.679842:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5485 00000100:00100000:2.0:1787829034.679842:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973067136:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829034.679856:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829034.679861:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829034.679866:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829034.679870:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973067264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829034.679877:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973067520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829034.679900:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973067392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829034.679938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973067520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829035.191425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829035.191430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787829035.447569:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829036.215419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829036.215422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829036.471490:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829037.239442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829037.239459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829037.495518:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829037.751753:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787829037.751815:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829037.751845:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829037.751893:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829037.751914:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829037.751927:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829037.751940:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829037.751950:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787829037.751960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829034/real 1787829037] req@ffff9c1a78a7a580 x1874668973066880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829037.751963:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787829037.751996:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973066880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.752009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.752016:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829037.752020:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973066880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829037.752029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829034/real 1787829037] req@ffff9c1a78a78f00 x1874668973067008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.752043:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973067008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.752051:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.752054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829037.752068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973067008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829037.752075:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829034/real 1787829037] req@ffff9c1a78a7b480 x1874668973067136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.752086:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973067136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.752095:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.752098:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829037.752102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973067136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829037.752109:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829034/real 1787829037] req@ffff9c1a78a79a40 x1874668973067264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.752120:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973067264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.752128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.752132:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829037.752135:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973067264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829037.752142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829034/real 1787829037] req@ffff9c1a78a78000 x1874668973066752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.752154:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973066752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.752161:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.752165:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829037.752168:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973066752:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829037.752217:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829034/real 1787829037] req@ffff9c1a78a792c0 x1874668973067392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.752230:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973067392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.752238:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.752241:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829037.752251:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973067392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829037.752258:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829034/real 1787829037] req@ffff9c1a78a7b840 x1874668973067520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829037.752272:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973067520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829037.752280:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829037.752284:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829037.752288:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973067520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829038.263472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829038.263472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829038.519505:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829039.287486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829039.287487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829039.543492:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829039.799510:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829039.799524:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829039.799540:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829039.799552:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5490 00000100:00080000:0.0:1787829039.799566:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829039.799576:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829039.799620:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973067648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829039.799653:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829039.799657:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829039.799659:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829039.799664:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5490 00000100:00080000:0.0:1787829039.799668:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829039.799673:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829039.799686:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829039.799699:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973067776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829039.799706:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973067648:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829039.799714:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829039.799719:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829039.799721:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829039.799726:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5490 00000100:00080000:0.0:1787829039.799730:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829039.799735:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829039.799741:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829039.799754:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973067904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829039.799766:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829039.799770:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829039.799772:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829039.799777:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5490 00000100:00080000:0.0:1787829039.799781:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829039.799786:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829039.799792:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829039.799802:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829039.799806:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973068032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829039.799817:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829039.799822:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829039.799824:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829039.799829:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5490 00000100:00080000:0.0:1787829039.799833:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829039.799837:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829039.799843:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829039.799856:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973068160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829039.799860:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829039/real 1787829039] req@ffff9c1a788621c0 x1874668973067648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829039.799877:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829039.799882:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829039.799885:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829039.799892:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5490 00000100:00100000:2.0:1787829039.799894:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973067648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829039.799898:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829039.799907:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829039.799916:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829039.799917:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829039.799925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829039.799930:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973067648:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829039.799945:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973068288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829039.799957:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829039.799961:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829039.799963:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829039.799968:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973067776:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829039.799971:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5490 00000100:00080000:0.0:1787829039.799979:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829039.799984:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829039.799990:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829039.800003:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973068416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829039.800007:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829039.800026:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829039/real 1787829039] req@ffff9c1a78860780 x1874668973067776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829039.800041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973067776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829039.800050:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829039.800053:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829039.800056:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973067776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829039.800063:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973067904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829039.800083:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829039.800095:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829039/real 1787829039] req@ffff9c1a78861e00 x1874668973067904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829039.800107:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973067904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829039.800115:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829039.800118:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829039.800122:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973067904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829039.800134:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973068032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829039.800154:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829039.800166:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829039/real 1787829039] req@ffff9c1a78862580 x1874668973068032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829039.800178:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973068032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829039.800186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829039.800189:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829039.800192:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973068032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829039.800197:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973068160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829039.800216:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829039.800227:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829039/real 1787829039] req@ffff9c1a78861680 x1874668973068160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829039.800238:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973068160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829039.800247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829039.800251:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829039.800254:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973068160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829039.800260:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973068288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829039.800281:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829039.800293:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829039/real 1787829039] req@ffff9c1a788630c0 x1874668973068288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829039.800305:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973068288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829039.800314:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829039.800317:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829039.800321:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973068288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829039.800386:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973068416:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829039.800413:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829039.800445:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829039/real 1787829039] req@ffff9c1a78863c00 x1874668973068416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829039.800457:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973068416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829039.800465:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829039.800468:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829039.800472:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973068416:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829040.311452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829040.311479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829040.567565:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829041.335455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829041.335502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829041.591490:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829042.359424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829042.359437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787829042.615506:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829043.383418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829043.383428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829043.639488:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829044.407421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829044.407427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829044.663516:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829044.919519:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829044.919530:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829044.919535:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829044.919545:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5495 00000100:00080000:3.0:1787829044.919558:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829044.919565:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829044.919610:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973068544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829044.919639:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829044.919642:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829044.919643:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829044.919646:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5495 00000100:00080000:3.0:1787829044.919648:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829044.919651:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829044.919661:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829044.919673:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973068672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829044.919679:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829044.919682:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829044.919683:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829044.919687:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5495 00000100:00080000:3.0:1787829044.919689:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829044.919691:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829044.919693:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829044.919700:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973068800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829044.919713:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973068544:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829044.919714:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829044.919726:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829044.919729:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829044.919733:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5495 00000100:00080000:3.0:1787829044.919736:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829044.919738:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829044.919742:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829044.919755:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973068928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829044.919761:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829044.919764:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829044.919765:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829044.919768:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5495 00000100:00080000:3.0:1787829044.919789:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829044.919791:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829044.919794:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829044.919803:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829044.919804:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973069056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829044.919816:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829044.919819:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829044.919820:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829044.919823:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5495 00000100:00080000:3.0:1787829044.919827:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829044.919831:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829044.919835:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829044.919845:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973069184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829044.919856:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829044.919858:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829044/real 1787829044] req@ffff9c1a7895de00 x1874668973068544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829044.919861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829044.919866:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829044.919878:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5495 00000100:00080000:3.0:1787829044.919887:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829044.919893:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829044.919903:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973068544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829044.919903:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829044.919917:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973069312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829044.919926:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829044.919935:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829044.919941:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973068544:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829044.919952:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973068672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829044.919978:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829044.919988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829044/real 1787829044] req@ffff9c1a7895f0c0 x1874668973068672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829044.919997:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973068672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829044.920003:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829044.920006:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829044.920008:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973068672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829044.920029:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973068800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829044.920041:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829044.920048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829044/real 1787829044] req@ffff9c1a7895f840 x1874668973068800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829044.920056:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973068800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829044.920061:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829044.920064:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829044.920066:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973068800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829044.920070:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973068928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829044.920080:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829044.920086:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829044/real 1787829044] req@ffff9c1a7895fc00 x1874668973068928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829044.920093:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973068928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829044.920099:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829044.920100:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829044.920102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973068928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829044.920106:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973069056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829044.920117:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829044.920122:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829044/real 1787829044] req@ffff9c1a7895e1c0 x1874668973069056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829044.920130:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973069056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829044.920135:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829044.920137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829044.920139:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973069056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829044.920143:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973069184:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829044.920154:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829044.920160:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829044/real 1787829044] req@ffff9c1a7895da40 x1874668973069184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829044.920167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973069184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829044.920172:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829044.920174:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829044.920176:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973069184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829044.920180:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973069312:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829044.920189:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829044.920194:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829044/real 1787829044] req@ffff9c1a7895cb40 x1874668973069312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829044.920207:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973069312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829044.920212:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829044.920214:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829044.920215:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973069312:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829045.431477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829045.431477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829045.687557:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829046.455454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829046.455454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829046.711479:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829047.479448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829047.479448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829047.735510:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829048.503425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829048.503459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829048.759524:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829049.527452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829049.527452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787829049.783539:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829050.039474:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829050.039485:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829050.039490:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829050.039501:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5500 00000100:00080000:3.0:1787829050.039520:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829050.039529:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829050.039581:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973069440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829050.039612:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829050.039615:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829050.039617:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829050.039621:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5500 00000100:00080000:3.0:1787829050.039624:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829050.039627:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829050.039637:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829050.039647:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973069568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829050.039655:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829050.039657:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829050.039659:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829050.039661:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5500 00000100:00080000:3.0:1787829050.039663:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829050.039665:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829050.039670:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829050.039672:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973069440:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829050.039679:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973069696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829050.039688:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829050.039691:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829050.039692:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829050.039695:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5500 00000100:00080000:3.0:1787829050.039698:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829050.039701:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829050.039704:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829050.039714:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973069824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829050.039720:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829050.039723:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829050.039724:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829050.039726:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5500 00000100:00080000:3.0:1787829050.039736:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829050.039738:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829050.039741:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829050.039749:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829050.039750:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973069952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829050.039759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829050.039761:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829050.039762:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829050.039765:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5500 00000100:00080000:3.0:1787829050.039767:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829050.039770:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829050.039773:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829050.039780:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973070080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829050.039786:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829050.039789:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829050.039790:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829050.039793:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5500 00000100:00080000:3.0:1787829050.039796:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829050.039799:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829050.039803:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829050.039813:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff9c1a7895c780 x1874668973069440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829050.039813:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973070208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829050.039850:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973069440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.039864:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.039870:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829050.039875:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973069440:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829050.039883:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973069568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829050.039905:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829050.039916:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff9c1a7895e940 x1874668973069568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.039924:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973069568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.039930:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.039932:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829050.039934:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973069568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829050.039957:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973069696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829050.039969:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829050.039975:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff9c1a7895c3c0 x1874668973069696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.039983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973069696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.039988:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.039990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829050.039992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973069696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829050.039996:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973069824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829050.040005:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829050.040011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff9c1a7895cf00 x1874668973069824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.040018:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973069824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.040023:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.040025:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829050.040028:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973069824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829050.040031:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973069952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829050.040040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829050.040046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff9c1a7895f480 x1874668973069952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.040053:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973069952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.040058:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.040059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829050.040061:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973069952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829050.040065:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973070080:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829050.040076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829050.040081:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff9c1a7895d2c0 x1874668973070080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.040088:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973070080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.040093:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.040095:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829050.040097:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973070080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829050.040101:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973070208:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829050.040110:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829050.040115:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829050/real 1787829050] req@ffff9c1a7895c000 x1874668973070208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829050.040122:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973070208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829050.040127:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829050.040129:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829050.040130:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973070208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829050.551430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829050.551433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829050.807520:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829051.575459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829051.575461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829051.831520:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829052.599412:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829052.599475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829052.855535:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829053.623464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829053.623464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829053.879572:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829054.647453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829054.647531:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829054.903496:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829055.159450:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829055.159458:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829055.159462:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829055.159470:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5505 00000100:00080000:1.0:1787829055.159479:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829055.159485:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829055.159514:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973070336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829055.159535:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829055.159538:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829055.159539:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829055.159542:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5505 00000100:00080000:1.0:1787829055.159545:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829055.159547:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829055.159554:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829055.159563:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973070464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829055.159570:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829055.159573:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829055.159574:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829055.159576:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5505 00000100:00080000:1.0:1787829055.159578:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829055.159580:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829055.159583:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829055.159590:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973070336:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829055.159593:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973070592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829055.159606:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829055.159608:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829055.159610:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829055.159612:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5505 00000100:00080000:1.0:1787829055.159614:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829055.159616:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829055.159619:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829055.159626:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973070720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829055.159632:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829055.159634:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829055.159635:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829055.159637:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5505 00000100:00080000:1.0:1787829055.159646:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829055.159649:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829055.159652:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829055.159661:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973070848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829055.159667:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829055.159672:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829055.159675:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829055.159677:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829055.159681:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5505 00000100:00080000:1.0:1787829055.159684:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829055.159692:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829055.159698:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829055.159718:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973070976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829055.159718:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff9c1a78a7bc00 x1874668973070336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829055.159754:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829055.159757:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829055.159759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829055.159764:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973070336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829055.159770:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5505 00000100:00080000:1.0:1787829055.159776:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829055.159782:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829055.159783:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829055.159787:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829055.159791:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829055.159797:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973070336:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829055.159800:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973071104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829055.159812:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973070464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829055.159840:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829055.159854:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff9c1a78a78780 x1874668973070464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829055.159869:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973070464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829055.159877:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829055.159881:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829055.159884:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973070464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829055.159908:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973070592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829055.159928:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829055.159939:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff9c1a78a7a940 x1874668973070592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829055.159950:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973070592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829055.159958:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829055.159961:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829055.159964:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973070592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829055.159971:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973070720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829055.159988:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829055.160000:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff9c1a78a783c0 x1874668973070720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829055.160011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973070720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829055.160020:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829055.160024:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829055.160027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973070720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829055.160034:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973070848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829055.160054:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829055.160065:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff9c1a78a7a1c0 x1874668973070848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829055.160077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973070848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829055.160085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829055.160089:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829055.160093:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973070848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829055.160109:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973070976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829055.160134:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829055.160145:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff9c1a78a7b0c0 x1874668973070976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829055.160157:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973070976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829055.160166:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829055.160169:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829055.160172:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973070976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829055.160179:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973071104:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829055.160195:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829055.160204:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829055/real 1787829055] req@ffff9c1a78a79e00 x1874668973071104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829055.160216:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973071104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829055.160225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829055.160228:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829055.160232:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973071104:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829055.671483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829055.671492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829055.927575:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829056.695402:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829056.695409:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787829056.951518:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829057.719453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829057.719469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829057.975520:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829058.743478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829058.743479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829058.999513:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829059.767435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829059.767459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829060.023478:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829060.279508:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829060.279519:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829060.279525:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829060.279536:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5510 00000100:00080000:0.0:1787829060.279549:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829060.279556:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829060.279600:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973071232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829060.279628:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829060.279631:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829060.279632:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829060.279636:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5510 00000100:00080000:0.0:1787829060.279638:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829060.279641:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829060.279651:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829060.279669:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973071360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829060.279678:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829060.279682:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829060.279688:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829060.279690:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973071232:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829060.279696:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5510 00000100:00080000:0.0:1787829060.279704:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829060.279709:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829060.279712:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829060.279724:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973071488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829060.279737:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829060.279740:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829060.279742:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829060.279746:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5510 00000100:00080000:0.0:1787829060.279749:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829060.279751:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829060.279754:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829060.279766:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973071616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829060.279774:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829060.279786:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829060.279793:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829060.279795:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829060.279800:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5510 00000100:00080000:0.0:1787829060.279804:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829060.279808:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829060.279813:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829060.279820:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff9c1a78860000 x1874668973071232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829060.279826:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973071744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829060.279873:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829060.279876:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829060.279881:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973071232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829060.279882:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829060.279897:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5510 00000100:00080000:0.0:1787829060.279907:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829060.279918:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829060.279922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829060.279925:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829060.279932:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829060.279939:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973071232:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829060.279942:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973071872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829060.279952:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829060.279955:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829060.279957:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829060.279961:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5510 00000100:00080000:0.0:1787829060.279965:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829060.279971:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829060.279976:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973071360:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787829060.279981:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829060.279991:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973072000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829060.280007:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829060.280027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff9c1a78863c00 x1874668973071360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829060.280043:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973071360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829060.280052:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829060.280055:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829060.280057:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973071360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829060.280063:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973071488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829060.280079:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829060.280087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff9c1a788630c0 x1874668973071488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829060.280095:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973071488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829060.280101:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829060.280103:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829060.280105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973071488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829060.280109:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973071616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829060.280119:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829060.280126:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff9c1a78861680 x1874668973071616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829060.280133:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973071616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829060.280139:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829060.280141:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829060.280143:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973071616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829060.280147:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973071744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829060.280157:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829060.280163:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff9c1a78862580 x1874668973071744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829060.280170:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973071744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829060.280176:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829060.280177:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829060.280179:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973071744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829060.280183:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973071872:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829060.280194:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829060.280200:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff9c1a78861e00 x1874668973071872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829060.280207:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973071872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829060.280213:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829060.280215:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829060.280217:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973071872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829060.280231:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973072000:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829060.280241:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829060.280247:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829060/real 1787829060] req@ffff9c1a78860780 x1874668973072000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829060.280254:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973072000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829060.280260:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829060.280262:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829060.280264:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973072000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829060.791497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829060.791497:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829061.047536:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829061.815424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829061.815424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829062.071486:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829062.839446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829062.839447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829063.095583:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829063.863432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829063.863434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829064.119601:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829064.887464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829064.887464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829065.143646:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829065.399464:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829065.399475:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829065.399479:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829065.399491:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5515 00000100:00080000:0.0:1787829065.399504:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829065.399514:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829065.399563:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973072128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829065.399598:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829065.399603:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829065.399606:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829065.399611:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5515 00000100:00080000:0.0:1787829065.399617:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829065.399623:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829065.399636:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829065.399654:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973072128:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829065.399660:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973072256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829065.399683:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829065.399695:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829065.399699:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829065.399706:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5515 00000100:00080000:0.0:1787829065.399715:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829065.399722:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829065.399729:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829065.399747:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973072384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829065.399761:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829065.399766:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829065.399768:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829065.399773:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5515 00000100:00080000:0.0:1787829065.399779:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829065.399784:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829065.399790:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829065.399798:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829065.399820:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973072512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829065.399837:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829065.399842:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829065.399844:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829065.399853:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5515 00000100:00100000:2.0:1787829065.399859:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff9c1a788621c0 x1874668973072128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829065.399860:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829065.399879:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829065.399889:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829065.399896:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973072128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829065.399908:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829065.399914:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973072640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829065.399916:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829065.399930:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973072128:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829065.399957:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829065.399961:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829065.399963:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829065.399968:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5515 00000100:00080000:0.0:1787829065.399973:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829065.399978:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973072256:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829065.399979:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829065.399988:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829065.400015:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973072768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829065.400019:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829065.400035:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff9c1a78863840 x1874668973072256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829065.400042:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829065.400048:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973072256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829065.400052:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829065.400060:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829065.400073:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5515 00000100:00080000:2.0:1787829065.400073:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829065.400083:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829065.400084:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829065.400092:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973072256:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829065.400093:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829065.400113:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829065.400116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973072384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829065.400134:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973072896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829065.400147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829065.400179:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff9c1a788612c0 x1874668973072384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829065.400206:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973072384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829065.400217:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829065.400222:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829065.400229:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973072384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829065.400239:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973072512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829065.400264:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829065.400285:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff9c1a78862940 x1874668973072512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829065.400300:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973072512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829065.400315:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829065.400317:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829065.400320:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973072512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829065.400325:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973072640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829065.400337:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829065.400384:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff9c1a78861a40 x1874668973072640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829065.400395:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973072640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829065.400411:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829065.400414:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829065.400417:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973072640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829065.400442:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973072768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829065.400468:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829065.400481:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff9c1a78860f00 x1874668973072768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829065.400492:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973072768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829065.400506:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829065.400509:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829065.400512:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973072768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829065.400519:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973072896:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829065.400539:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829065.400548:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829065/real 1787829065] req@ffff9c1a78862d00 x1874668973072896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829065.400556:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973072896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829065.400561:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829065.400563:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829065.400565:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973072896:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829065.911451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829065.911499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829066.167642:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829066.935449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829066.935463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829067.191557:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829067.959440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829067.959448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829068.215629:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829068.983547:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829068.983547:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829069.239624:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829070.007416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829070.007468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829070.263541:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829070.519462:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829070.519470:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829070.519474:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829070.519483:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5520 00000100:00080000:2.0:1787829070.519492:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829070.519498:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829070.519535:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973073024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829070.519570:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829070.519574:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829070.519575:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829070.519586:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5520 00000100:00080000:2.0:1787829070.519591:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829070.519595:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829070.519604:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829070.519615:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973073152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829070.519626:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829070.519630:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973073024:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829070.519630:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829070.519636:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829070.519649:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5520 00000100:00080000:2.0:1787829070.519651:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829070.519653:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829070.519657:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829070.519669:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973073280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829070.519677:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829070.519680:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829070.519682:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829070.519684:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5520 00000100:00080000:2.0:1787829070.519686:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829070.519689:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829070.519694:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829070.519701:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973073408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829070.519709:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829070.519711:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829070.519713:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829070.519715:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5520 00000100:00080000:2.0:1787829070.519718:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829070.519726:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829070.519729:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829070.519735:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829070.519744:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973073536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829070.519752:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829070.519755:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829070.519757:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829070.519760:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5520 00000100:00080000:2.0:1787829070.519763:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829070.519767:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829070.519773:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829070.519780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829070/real 1787829070] req@ffff9c1a49d8da40 x1874668973073024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829070.519785:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973073664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829070.519845:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829070.519848:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829070.519850:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829070.519856:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5520 00000100:00100000:1.0:1787829070.519858:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973073024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829070.519862:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829070.519875:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829070.519889:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829070.519895:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829070.519900:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829070.519900:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973073792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829070.519915:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973073024:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829070.519951:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973073152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829070.519971:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829070.519980:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829070/real 1787829070] req@ffff9c1a49d8de00 x1874668973073152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829070.519995:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973073152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829070.520001:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829070.520003:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829070.520005:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973073152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829070.520018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973073280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829070.520032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829070.520044:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829070/real 1787829070] req@ffff9c1a49d8c780 x1874668973073280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829070.520057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973073280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829070.520067:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829070.520070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829070.520074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973073280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829070.520090:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973073408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829070.520111:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829070.520120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829070/real 1787829070] req@ffff9c1a49d8c000 x1874668973073408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829070.520128:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973073408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829070.520139:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829070.520141:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829070.520144:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973073408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829070.520148:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973073536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829070.520159:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829070.520165:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829070/real 1787829070] req@ffff9c1a49d8cf00 x1874668973073536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829070.520172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973073536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829070.520178:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829070.520180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829070.520181:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973073536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829070.520186:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973073664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829070.520198:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829070.520204:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829070/real 1787829070] req@ffff9c1a49d8cb40 x1874668973073664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829070.520211:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973073664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829070.520216:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829070.520218:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829070.520220:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973073664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829070.520224:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973073792:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829070.520234:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829070.520240:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829070/real 1787829070] req@ffff9c1a49d8f0c0 x1874668973073792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829070.520246:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973073792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829070.520251:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829070.520253:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829070.520255:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973073792:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829071.031502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829071.031512:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829071.287555:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829072.055417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829072.055494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829072.311588:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829073.079514:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829073.079519:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829073.335516:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829074.103394:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829074.103411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829074.359462:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829075.127458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829075.127472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829075.383535:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829075.639485:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829075.639511:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829075.639514:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829075.639522:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5525 00000100:00080000:1.0:1787829075.639531:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829075.639537:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829075.639592:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973073920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829075.639636:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829075.639639:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829075.639641:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829075.639645:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5525 00000100:00080000:1.0:1787829075.639649:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829075.639653:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829075.639662:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829075.639675:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973074048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829075.639684:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973073920:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829075.639687:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829075.639694:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829075.639697:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829075.639701:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5525 00000100:00080000:1.0:1787829075.639704:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829075.639706:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829075.639710:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829075.639720:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973074176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829075.639728:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829075.639731:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829075.639733:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829075.639736:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5525 00000100:00080000:1.0:1787829075.639739:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829075.639742:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829075.639746:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829075.639761:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973074304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829075.639767:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829075.639775:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829075.639778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829075.639780:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829075.639784:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5525 00000100:00080000:1.0:1787829075.639788:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829075.639791:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829075.639794:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829075.639803:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973074432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829075.639810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829075.639812:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829075.639814:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829075.639817:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5525 00000100:00080000:1.0:1787829075.639836:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829075.639840:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829075.639843:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829075.639848:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829075/real 1787829075] req@ffff9c1a78a79e00 x1874668973073920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829075.639853:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973074560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829075.639881:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829075.639906:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829075.639908:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829075.639908:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973073920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829075.639917:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5525 00000100:00080000:1.0:1787829075.639922:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829075.639928:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829075.639933:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829075.639934:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829075.639943:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829075.639946:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973074688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829075.639954:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973073920:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829075.639988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973074048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829075.640017:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829075.640046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829075/real 1787829075] req@ffff9c1a78a7b0c0 x1874668973074048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829075.640059:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973074048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829075.640074:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829075.640077:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829075.640081:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973074048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829075.640087:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973074176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829075.640119:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829075.640129:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829075/real 1787829075] req@ffff9c1a78a7a1c0 x1874668973074176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829075.640142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973074176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829075.640152:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829075.640156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829075.640160:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973074176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829075.640181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973074304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829075.640200:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829075.640212:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829075/real 1787829075] req@ffff9c1a78a783c0 x1874668973074304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829075.640225:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973074304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829075.640251:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829075.640255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829075.640259:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973074304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829075.640266:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973074432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829075.640301:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829075.640314:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829075/real 1787829075] req@ffff9c1a78a7a940 x1874668973074432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829075.640327:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973074432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829075.640337:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829075.640340:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829075.640365:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973074432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829075.640374:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973074560:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829075.640398:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829075.640411:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829075/real 1787829075] req@ffff9c1a78a78780 x1874668973074560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829075.640424:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973074560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829075.640433:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829075.640437:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829075.640441:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973074560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829075.640447:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973074688:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829075.640469:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829075.640482:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829075/real 1787829075] req@ffff9c1a78a7bc00 x1874668973074688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829075.640496:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973074688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829075.640526:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829075.640530:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829075.640533:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973074688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829076.151514:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829076.151514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829076.407540:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829077.175495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829077.175500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829077.431511:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829078.199461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829078.199461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829078.455531:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829079.223503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829079.223510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829079.479569:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829080.247444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829080.247444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829080.503580:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829080.759548:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829080.759563:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829080.759585:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829080.759600:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5531 00000100:00080000:0.0:1787829080.759617:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829080.759629:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829080.759677:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973074816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829080.759722:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829080.759729:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829080.759732:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829080.759741:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5531 00000100:00080000:0.0:1787829080.759748:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829080.759758:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829080.759773:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829080.759784:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973074816:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829080.759823:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973074944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829080.759844:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829080.759849:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829080.759853:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829080.759860:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5531 00000100:00080000:0.0:1787829080.759867:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829080.759873:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829080.759882:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829080.759899:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973075072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829080.759925:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829080.759930:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829080.759933:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829080.759938:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5531 00000100:00080000:0.0:1787829080.759944:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829080.759945:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829080.759958:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829080.759967:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829080.759989:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973075200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829080.759999:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829080/real 1787829080] req@ffff9c1a78860b40 x1874668973074816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829080.760014:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829080.760022:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829080.760028:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973074816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829080.760029:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829080.760046:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5531 00000100:00080000:0.0:1787829080.760053:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829080.760057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829080.760063:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829080.760068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829080.760076:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973074816:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829080.760078:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829080.760122:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973075328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829080.760128:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973074944:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829080.760155:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829080.760162:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829080.760166:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829080.760172:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829080.760180:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5531 00000100:00080000:0.0:1787829080.760191:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829080.760194:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829080/real 1787829080] req@ffff9c1a78862d00 x1874668973074944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829080.760204:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829080.760222:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829080.760223:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973074944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829080.760272:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829080.760279:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973075456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829080.760281:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829080.760293:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973074944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829080.760317:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973075072:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829080.760327:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829080.760336:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829080.760341:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829080.760396:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829080.760418:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829080/real 1787829080] req@ffff9c1a78860f00 x1874668973075072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829080.760432:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5531 00000100:00080000:0.0:1787829080.760445:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829080.760445:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973075072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829080.760477:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829080.760481:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829080.760488:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787829080.760491:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829080.760493:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973075072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829080.760506:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973075200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829080.760523:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973075584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829080.760566:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829080.760586:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829080/real 1787829080] req@ffff9c1a78861a40 x1874668973075200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829080.760600:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973075200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829080.760613:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829080.760617:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829080.760623:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973075200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829080.760654:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973075328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829080.760676:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829080.760688:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829080/real 1787829080] req@ffff9c1a78862940 x1874668973075328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829080.760701:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973075328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829080.760710:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829080.760713:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829080.760717:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973075328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829080.760723:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973075456:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829080.760741:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829080.760750:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829080/real 1787829080] req@ffff9c1a788612c0 x1874668973075456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829080.760762:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973075456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829080.760771:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829080.760774:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829080.760777:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973075456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829080.760782:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973075584:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829080.760798:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829080.760826:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829080/real 1787829080] req@ffff9c1a78863840 x1874668973075584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829080.760837:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973075584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829080.760844:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829080.760847:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829080.760850:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973075584:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829081.271500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829081.271504:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829081.527487:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829082.295486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829082.295491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829082.551476:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829083.319437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829083.319437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829083.575519:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829084.343444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829084.343450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829084.599509:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829085.367451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829085.367451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829085.623472:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829085.879470:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829085.879477:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829085.879481:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829085.879489:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5536 00000100:00080000:2.0:1787829085.879499:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829085.879504:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829085.879537:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973075712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829085.879562:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829085.879564:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829085.879566:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829085.879569:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5536 00000100:00080000:2.0:1787829085.879572:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829085.879574:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829085.879582:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829085.879592:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973075840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829085.879599:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829085.879602:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829085.879603:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829085.879606:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5536 00000100:00080000:2.0:1787829085.879608:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829085.879613:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973075712:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829085.879613:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829085.879618:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829085.879626:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973075968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829085.879635:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829085.879637:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829085.879639:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829085.879641:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5536 00000100:00080000:2.0:1787829085.879643:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829085.879645:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829085.879648:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829085.879656:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973076096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829085.879662:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829085.879665:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829085.879666:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829085.879669:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5536 00000100:00080000:2.0:1787829085.879672:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829085.879677:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787829085.879679:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787829085.879681:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829085.879689:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973076224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829085.879697:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829085.879700:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829085.879701:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829085.879703:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5536 00000100:00080000:2.0:1787829085.879705:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829085.879708:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829085.879711:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829085.879718:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973076352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829085.879724:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829085.879728:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829085.879729:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829085/real 1787829085] req@ffff9c1a49d8e1c0 x1874668973075712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829085.879730:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829085.879737:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5536 00000100:00080000:2.0:1787829085.879741:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829085.879747:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829085.879752:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829085.879762:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973076480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829085.879769:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973075712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829085.879784:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829085.879788:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829085.879792:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973075712:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829085.879798:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973075840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829085.879821:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829085.879836:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829085/real 1787829085] req@ffff9c1a49d8c3c0 x1874668973075840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829085.879850:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973075840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829085.879860:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829085.879863:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829085.879866:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973075840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829085.879887:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973075968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829085.879904:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829085.879928:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829085/real 1787829085] req@ffff9c1a49d8e580 x1874668973075968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829085.879949:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973075968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829085.879958:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829085.879961:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829085.879965:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973075968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829085.879971:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973076096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829085.879992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829085.880004:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829085/real 1787829085] req@ffff9c1a49d8fc00 x1874668973076096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829085.880015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973076096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829085.880024:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829085.880027:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829085.880030:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973076096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829085.880037:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973076224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829085.880057:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829085.880066:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829085/real 1787829085] req@ffff9c1a49d8ed00 x1874668973076224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829085.880074:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973076224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829085.880084:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829085.880086:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829085.880088:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973076224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829085.880092:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973076352:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829085.880105:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829085.880110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829085/real 1787829085] req@ffff9c1a49d8d680 x1874668973076352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829085.880118:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973076352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829085.880123:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829085.880124:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829085.880126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973076352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829085.880130:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973076480:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829085.880139:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829085.880145:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829085/real 1787829085] req@ffff9c1a49d8f0c0 x1874668973076480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829085.880152:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973076480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829085.880157:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829085.880159:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829085.880161:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973076480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829086.391415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829086.391426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829086.647519:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829087.415426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829087.415465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829087.671522:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829088.439474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829088.439475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829088.695493:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829089.463437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829089.463437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829089.719519:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829090.487452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829090.487452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829090.743563:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829090.999482:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829090.999491:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829090.999507:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829090.999515:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5541 00000100:00080000:0.0:1787829090.999526:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829090.999535:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829090.999618:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973076608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829090.999642:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829090.999645:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829090.999647:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829090.999651:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5541 00000100:00080000:0.0:1787829090.999654:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829090.999658:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829090.999676:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829090.999687:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973076736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829090.999696:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829090.999698:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829090.999700:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829090.999705:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973076608:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829090.999706:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5541 00000100:00080000:0.0:1787829090.999711:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829090.999717:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829090.999720:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829090.999727:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973076864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829090.999739:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829090.999741:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829090.999742:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829090.999744:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5541 00000100:00080000:0.0:1787829090.999746:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829090.999748:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829090.999751:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829090.999757:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973076992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829090.999763:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829090.999766:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829090.999768:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829090.999770:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5541 00000100:00080000:0.0:1787829090.999772:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829090.999774:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829090.999777:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829090.999785:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973077120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829090.999791:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829090.999793:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829090.999795:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829090.999797:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5541 00000100:00080000:0.0:1787829090.999800:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829090.999804:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787829090.999808:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787829090.999810:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829090.999820:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973077248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829090.999828:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829090.999832:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829090.999834:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829090.999837:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5541 00000100:00080000:0.0:1787829090.999842:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829090.999846:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829090.999853:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829090.999854:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829090/real 1787829090] req@ffff9c1a788621c0 x1874668973076608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829090.999871:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973077376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829090.999874:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973076608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829090.999905:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829090.999909:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829090.999913:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973076608:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829090.999943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973076736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829090.999961:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829090.999971:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829090/real 1787829090] req@ffff9c1a78860780 x1874668973076736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829090.999979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973076736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829090.999990:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829090.999993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829090.999995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973076736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829090.999999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973076864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829091.000011:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829091.000024:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829090/real 1787829091] req@ffff9c1a78861e00 x1874668973076864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829091.000044:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973076864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829091.000049:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829091.000051:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829091.000053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973076864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829091.000057:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973076992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829091.000069:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829091.000076:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829090/real 1787829091] req@ffff9c1a78862580 x1874668973076992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829091.000083:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973076992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829091.000088:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829091.000090:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829091.000105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973076992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829091.000108:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973077120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829091.000119:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829091.000125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829090/real 1787829091] req@ffff9c1a78861680 x1874668973077120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829091.000132:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973077120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829091.000137:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829091.000138:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829091.000140:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973077120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829091.000144:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973077248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829091.000155:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829091.000161:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829090/real 1787829091] req@ffff9c1a788630c0 x1874668973077248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829091.000168:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973077248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829091.000173:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829091.000189:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829091.000191:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973077248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829091.000196:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973077376:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829091.000222:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829091.000230:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829090/real 1787829091] req@ffff9c1a78863c00 x1874668973077376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829091.000238:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973077376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829091.000243:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829091.000245:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829091.000247:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973077376:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829091.511441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829091.511470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829091.767548:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829092.535458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829092.535458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829092.791546:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829093.559400:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829093.559419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829093.816687:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829094.583481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829094.583481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829094.839540:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829095.607448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829095.607448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829095.863462:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829096.119450:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829096.119458:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829096.119462:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829096.119470:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5546 00000100:00080000:1.0:1787829096.119479:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829096.119484:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829096.119516:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973077504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829096.119553:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829096.119557:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829096.119560:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829096.119564:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5546 00000100:00080000:1.0:1787829096.119568:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829096.119573:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829096.119584:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829096.119597:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973077504:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829096.119603:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973077632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829096.119618:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829096.119622:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829096.119624:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829096.119644:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5546 00000100:00080000:1.0:1787829096.119647:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829096.119651:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829096.119656:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829096.119668:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973077760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829096.119679:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829096.119680:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829096.119686:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829096.119688:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829096.119711:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5546 00000100:00080000:1.0:1787829096.119716:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829096.119721:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829096.119727:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829096.119737:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff9c1a78a7b840 x1874668973077504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829096.119742:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973077888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829096.119772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829096.119775:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829096.119777:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829096.119781:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973077504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829096.119782:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5546 00000100:00080000:1.0:1787829096.119789:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829096.119812:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829096.119817:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829096.119818:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829096.119826:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829096.119831:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973078016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829096.119832:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973077504:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829096.119845:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829096.119848:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829096.119849:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829096.119853:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5546 00000100:00080000:1.0:1787829096.119855:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829096.119858:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829096.119861:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973077632:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787829096.119864:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829096.119876:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973078144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829096.119884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829096.119888:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787829096.119889:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829096.119890:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829096.119897:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5546 00000100:00080000:1.0:1787829096.119901:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829096.119905:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff9c1a78a792c0 x1874668973077632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829096.119906:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829096.119913:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829096.119924:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973077632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829096.119943:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973078272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829096.119960:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829096.119973:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829096.119978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973077632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829096.119985:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973077760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829096.120001:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829096.120026:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff9c1a78a78000 x1874668973077760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829096.120035:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973077760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829096.120041:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829096.120043:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829096.120045:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973077760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829096.120049:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973077888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829096.120060:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829096.120066:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff9c1a78a79a40 x1874668973077888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829096.120090:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973077888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829096.120096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829096.120098:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829096.120101:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973077888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829096.120104:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973078016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829096.120115:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829096.120121:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff9c1a78a7b480 x1874668973078016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829096.120134:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973078016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829096.120157:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829096.120160:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829096.120162:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973078016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829096.120176:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973078144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829096.120208:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829096.120216:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff9c1a78a78f00 x1874668973078144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829096.120224:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973078144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829096.120230:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829096.120232:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829096.120234:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973078144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829096.120237:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973078272:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829096.120246:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829096.120252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829096/real 1787829096] req@ffff9c1a78a7a580 x1874668973078272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829096.120259:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973078272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829096.120269:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829096.120271:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829096.120274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973078272:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829096.631435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829096.631435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829096.887511:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829097.655450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829097.655450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829097.911499:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829098.679412:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829098.679835:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829098.935534:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829099.703478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829099.703514:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829099.959536:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829100.727427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829100.727427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829100.983529:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829101.239485:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829101.239496:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829101.239501:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829101.239512:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5551 00000100:00080000:3.0:1787829101.239525:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829101.239533:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829101.239570:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973078400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829101.239601:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829101.239604:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829101.239606:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829101.239638:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5551 00000100:00080000:3.0:1787829101.239643:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829101.239648:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829101.239658:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829101.239675:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973078528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829101.239685:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829101.239689:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829101.239691:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829101.239695:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5551 00000100:00080000:3.0:1787829101.239699:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829101.239703:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829101.239708:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829101.239720:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973078656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829101.239732:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829101.239736:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829101.239737:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829101.239740:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5551 00000100:00080000:3.0:1787829101.239743:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829101.239746:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829101.239750:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829101.239758:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973078784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829101.239764:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829101.239767:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829101.239768:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829101.239770:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5551 00000100:00080000:3.0:1787829101.239772:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829101.239774:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829101.239776:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829101.239782:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973078912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829101.239787:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829101.239789:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829101.239790:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829101.239792:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5551 00000100:00080000:3.0:1787829101.239794:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829101.239796:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829101.239799:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829101.239804:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973079040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829101.239809:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829101.239811:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829101.239812:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829101.239814:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5551 00000100:00080000:3.0:1787829101.239816:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829101.239819:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829101.239823:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829101.239832:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973079168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829101.240158:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973078400:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829101.240239:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973078528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829101.240260:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973078656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829101.240278:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973078784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829101.240297:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973078912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829101.240325:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973079040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829101.240375:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973079168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829101.751465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829101.751465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829102.007518:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829102.775430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829102.775436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829103.031452:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829103.799444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829103.799444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829104.055491:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829104.311641:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787829104.311702:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829104.311717:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829104.311767:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829104.311776:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829104.311782:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829104.311788:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829104.311798:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829104.311809:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787829104.311823:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829104] req@ffff9c1a7895e580 x1874668973078400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.311852:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973078400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.311865:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.311870:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829104.311874:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973078400:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829104.311883:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829104] req@ffff9c1a7895c000 x1874668973078528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.311898:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973078528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.311907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.311911:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829104.311915:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973078528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829104.311921:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829104] req@ffff9c1a7895d2c0 x1874668973078656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.311944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973078656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.311953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.311956:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829104.311959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973078656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829104.311966:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829104] req@ffff9c1a7895f480 x1874668973078784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.311978:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973078784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.311987:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.311991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829104.311994:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973078784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829104.312001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829104] req@ffff9c1a7895cf00 x1874668973078912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.312015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973078912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.312024:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.312028:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829104.312032:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973078912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829104.312040:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829104] req@ffff9c1a7895c3c0 x1874668973079040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.312053:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973079040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.312063:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.312067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829104.312071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973079040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829104.312078:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829101/real 1787829104] req@ffff9c1a7895e940 x1874668973079168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.312109:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973079168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.312119:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.312122:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829104.312126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973079168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829104.823436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829104.823437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829105.079474:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829105.847430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829105.847430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829106.103540:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829106.359456:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829106.359465:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829106.359469:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829106.359477:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5556 00000100:00080000:3.0:1787829106.359486:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829106.359492:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829106.359523:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973079296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829106.359546:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829106.359549:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829106.359550:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829106.359553:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5556 00000100:00080000:3.0:1787829106.359555:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829106.359557:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829106.359565:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829106.359574:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973079424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829106.359582:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829106.359585:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829106.359588:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829106.359589:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973079296:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829106.359594:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5556 00000100:00080000:3.0:1787829106.359601:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829106.359605:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829106.359610:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829106.359623:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973079552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829106.359634:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829106.359638:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829106.359640:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829106.359644:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5556 00000100:00080000:3.0:1787829106.359649:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829106.359653:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829106.359657:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829106.359661:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829106.359672:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973079680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829106.359678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829106.359681:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829106.359682:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829106.359684:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5556 00000100:00080000:3.0:1787829106.359686:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829106.359688:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829106.359691:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829106.359696:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973079808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829106.359702:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829106.359704:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829106.359705:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829106.359708:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5556 00000100:00080000:3.0:1787829106.359709:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829106.359711:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829106.359714:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829106.359719:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973079936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829106.359728:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829106.359730:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829106.359732:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829106.359735:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5556 00000100:00080000:3.0:1787829106.359738:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829106.359741:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829106.359746:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829106.359756:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973080064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829106.361302:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829106] req@ffff9c1a7895c780 x1874668973079296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829106.361326:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973079296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829106.361337:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829106.361342:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829106.361362:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973079296:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829106.361391:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973079424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829106.361434:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829106.361447:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829106] req@ffff9c1a7895cb40 x1874668973079424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829106.361462:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973079424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829106.361471:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829106.361475:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829106.361478:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973079424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829106.361485:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973079552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829106.361507:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829106.361519:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829106] req@ffff9c1a7895da40 x1874668973079552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829106.361529:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973079552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829106.361537:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829106.361541:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829106.361544:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973079552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829106.361551:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973079680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829106.361572:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829106.361585:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829106] req@ffff9c1a7895e1c0 x1874668973079680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829106.361605:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973079680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829106.361613:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829106.361616:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829106.361619:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973079680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829106.361626:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973079808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829106.361645:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829106.361657:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829106] req@ffff9c1a7895fc00 x1874668973079808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829106.361669:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973079808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829106.361678:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829106.361681:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829106.361684:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973079808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829106.361691:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973079936:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829106.361719:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829106.361731:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829106] req@ffff9c1a7895f840 x1874668973079936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829106.361742:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973079936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829106.361757:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829106.361760:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829106.361763:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973079936:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829106.361770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973080064:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829106.361790:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829106.361801:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829106/real 1787829106] req@ffff9c1a7895f0c0 x1874668973080064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829106.361813:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973080064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829106.361823:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829106.361826:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829106.361829:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973080064:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829106.871439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829106.871451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829107.127566:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829107.895439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829107.895444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829108.151618:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829108.919449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829108.919450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829109.175495:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829109.943449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829109.943457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829110.199531:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829110.967422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829110.967422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829111.223499:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829111.479419:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829111.479427:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829111.479431:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829111.479439:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5561 00000100:00080000:3.0:1787829111.479450:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829111.479456:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829111.479489:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973080192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829111.479519:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829111.479523:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829111.479525:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829111.479529:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5561 00000100:00080000:3.0:1787829111.479533:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829111.479538:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829111.479548:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829111.479561:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973080192:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829111.479565:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973080320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829111.479575:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829111.479577:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829111.479579:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829111.479582:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5561 00000100:00080000:3.0:1787829111.479585:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829111.479588:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829111.479591:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829111.479599:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973080448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829111.479606:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829111.479609:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829111.479610:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829111.479613:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5561 00000100:00080000:3.0:1787829111.479615:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829111.479617:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829111.479620:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829111.479626:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973080576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829111.479631:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829111.479634:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829111.479635:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829111.479637:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5561 00000100:00080000:3.0:1787829111.479638:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829111.479640:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829111.479642:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829111.479647:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973080704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829111.479652:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829111.479654:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829111.479655:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829111.479657:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5561 00000100:00080000:3.0:1787829111.479658:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829111.479660:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829111.479665:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829111.479665:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829111.479675:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973080832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829111.479681:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829111.479684:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829111.479685:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829111.479687:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5561 00000100:00080000:3.0:1787829111.479689:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829111.479691:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829111.479694:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829111.479701:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973080960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829111.479712:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829111/real 1787829111] req@ffff9c1a7895de00 x1874668973080192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829111.479733:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973080192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829111.479742:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829111.479746:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829111.479750:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973080192:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829111.479770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973080320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829111.479789:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829111.479799:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829111/real 1787829111] req@ffff9c1a7895f0c0 x1874668973080320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829111.479807:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973080320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829111.479813:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829111.479815:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829111.479818:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973080320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829111.479821:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973080448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829111.479832:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829111.479838:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829111/real 1787829111] req@ffff9c1a7895f840 x1874668973080448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829111.479846:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973080448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829111.479851:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829111.479852:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829111.479854:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973080448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829111.479858:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973080576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829111.479867:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829111.479874:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829111/real 1787829111] req@ffff9c1a7895fc00 x1874668973080576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829111.479881:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973080576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829111.479887:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829111.479888:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829111.479890:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973080576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829111.479893:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973080704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829111.479903:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829111.479909:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829111/real 1787829111] req@ffff9c1a7895e1c0 x1874668973080704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829111.479917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973080704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829111.479922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829111.479923:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829111.479925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973080704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829111.479928:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973080832:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829111.479938:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829111.479944:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829111/real 1787829111] req@ffff9c1a7895da40 x1874668973080832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829111.479951:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973080832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829111.479957:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829111.479959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829111.479961:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973080832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829111.479964:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973080960:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829111.479973:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829111.479978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829111/real 1787829111] req@ffff9c1a7895cb40 x1874668973080960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829111.479985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973080960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829111.479990:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829111.479993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829111.479995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973080960:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829111.991474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829111.991539:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829112.247518:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829113.015401:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829113.015403:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829113.271502:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829114.039456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829114.039456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829114.295517:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829115.063445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829115.063445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829115.319470:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829116.087450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829116.087450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829116.343525:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829116.599440:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829116.599448:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829116.599451:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829116.599459:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5566 00000100:00080000:3.0:1787829116.599469:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829116.599474:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829116.599506:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973081088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829116.599538:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829116.599551:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829116.599553:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829116.599559:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5566 00000100:00080000:3.0:1787829116.599563:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829116.599568:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829116.599579:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829116.599588:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973081088:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829116.599597:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973081216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829116.599609:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829116.599614:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829116.599616:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829116.599620:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5566 00000100:00080000:3.0:1787829116.599638:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829116.599643:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829116.599649:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829116.599651:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829116.599663:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973081344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829116.599672:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829116.599674:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829116.599675:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829116.599678:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5566 00000100:00080000:3.0:1787829116.599681:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829116.599684:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829116.599687:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829116.599700:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973081472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829116.599701:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829116/real 1787829116] req@ffff9c1a7895c780 x1874668973081088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829116.599724:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829116.599727:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829116.599729:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829116.599731:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973081088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829116.599735:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5566 00000100:00080000:3.0:1787829116.599741:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829116.599747:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829116.599752:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829116.599752:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829116.599759:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829116.599763:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973081600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829116.599764:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973081088:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829116.599777:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829116.599779:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829116.599781:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829116.599784:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5566 00000100:00080000:3.0:1787829116.599786:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829116.599789:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829116.599792:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829116.599793:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973081216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829116.599808:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973081728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829116.599817:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829116.599821:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829116.599822:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829116.599825:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829116.599830:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5566 00000100:00080000:3.0:1787829116.599834:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829116.599837:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829116/real 1787829116] req@ffff9c1a7895e940 x1874668973081216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829116.599840:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829116.599846:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829116.599857:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973081216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829116.599861:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973081856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829116.599877:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829116.599883:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829116.599887:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973081216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829116.599892:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973081344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829116.599906:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829116.599914:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829116/real 1787829116] req@ffff9c1a7895c3c0 x1874668973081344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829116.599922:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973081344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829116.599928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829116.599931:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829116.599933:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973081344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829116.599937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973081472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829116.599948:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829116.599954:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829116/real 1787829116] req@ffff9c1a7895cf00 x1874668973081472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829116.599961:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973081472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829116.599967:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829116.599968:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829116.599970:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973081472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829116.599974:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973081600:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829116.599983:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829116.599989:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829116/real 1787829116] req@ffff9c1a7895f480 x1874668973081600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829116.600011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973081600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829116.600017:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829116.600020:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829116.600022:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973081600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829116.600035:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973081728:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829116.600051:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829116.600058:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829116/real 1787829116] req@ffff9c1a7895d2c0 x1874668973081728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829116.600066:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973081728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829116.600073:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829116.600074:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829116.600076:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973081728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829116.600080:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973081856:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829116.600089:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829116.600096:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829116/real 1787829116] req@ffff9c1a7895c000 x1874668973081856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829116.600108:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973081856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829116.600116:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829116.600120:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829116.600123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973081856:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829117.111452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829117.111452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829117.367471:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829118.135465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829118.135465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829118.391506:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829119.159455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829119.159455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829119.415486:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829120.183456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829120.183466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829120.439512:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829121.207466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829121.207466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829121.463524:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829121.719432:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829121.719441:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829121.719445:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829121.719453:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5572 00000100:00080000:3.0:1787829121.719463:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829121.719469:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829121.719503:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973081984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829121.719544:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829121.719547:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829121.719549:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829121.719552:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5572 00000100:00080000:3.0:1787829121.719555:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829121.719558:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829121.719566:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829121.719594:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973082112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829121.719603:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829121.719607:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829121.719609:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829121.719612:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5572 00000100:00080000:3.0:1787829121.719616:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829121.719620:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973081984:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829121.719621:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829121.719628:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829121.719654:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973082240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829121.719662:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829121.719664:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829121.719665:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829121.719668:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5572 00000100:00080000:3.0:1787829121.719670:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829121.719673:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829121.719677:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829121.719689:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973082368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829121.719699:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829121.719703:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829121.719722:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829121.719727:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829121.719733:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5572 00000100:00080000:3.0:1787829121.719748:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829121.719754:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829121.719760:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829121.719761:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829121/real 1787829121] req@ffff9c1a7895e580 x1874668973081984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829121.719774:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973082496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829121.719793:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973081984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829121.719793:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829121.719806:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829121.719811:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829121.719815:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829121.719820:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5572 00000100:00080000:2.0:1787829121.719823:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829121.719829:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829121.719830:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973081984:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829121.719840:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829121.719843:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973082112:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829121.719850:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829121.719869:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973082624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829121.719873:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829121.719884:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829121.719890:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829121.719891:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829121/real 1787829121] req@ffff9c1a7895c000 x1874668973082112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829121.719895:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829121.719904:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5572 00000100:00080000:3.0:1787829121.719910:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829121.719917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973082112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829121.719917:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829121.719926:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829121.719932:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829121.719937:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829121.719941:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973082752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829121.719945:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973082112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829121.720011:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973082240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829121.720029:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829121.720037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829121/real 1787829121] req@ffff9c1a7895d2c0 x1874668973082240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829121.720063:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973082240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829121.720075:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829121.720078:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829121.720081:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973082240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829121.720085:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973082368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829121.720100:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829121.720107:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829121/real 1787829121] req@ffff9c1a7895f480 x1874668973082368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829121.720134:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973082368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829121.720140:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829121.720142:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829121.720144:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973082368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829121.720148:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973082496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829121.720161:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829121.720168:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829121/real 1787829121] req@ffff9c1a7895cf00 x1874668973082496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829121.720177:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973082496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829121.720182:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829121.720184:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829121.720187:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973082496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829121.720191:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973082624:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829121.720203:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829121.720209:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829121/real 1787829121] req@ffff9c1a7895c3c0 x1874668973082624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829121.720217:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973082624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829121.720223:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829121.720225:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829121.720227:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973082624:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829121.720231:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973082752:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829121.720265:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829121.720276:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829121/real 1787829121] req@ffff9c1a7895e940 x1874668973082752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829121.720288:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973082752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829121.720298:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829121.720301:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829121.720305:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973082752:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829122.231448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829122.231451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829122.487555:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829123.255437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829123.255437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829123.511551:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829124.279432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829124.279432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829124.535454:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829125.303487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829125.303498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829125.559547:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829126.327418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829126.327453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829126.583463:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829126.839435:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829126.839445:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829126.839448:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829126.839457:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5577 00000100:00080000:3.0:1787829126.839466:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829126.839473:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829126.839521:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973082880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829126.839548:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829126.839552:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829126.839554:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829126.839559:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5577 00000100:00080000:3.0:1787829126.839564:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829126.839570:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829126.839580:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829126.839588:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973082880:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829126.839595:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973083008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829126.839611:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829126.839613:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829126.839615:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829126.839618:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5577 00000100:00080000:3.0:1787829126.839622:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829126.839624:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829126.839628:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829126.839636:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973083136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829126.839644:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829126.839646:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829126.839647:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829126.839649:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5577 00000100:00080000:3.0:1787829126.839677:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829126.839679:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829126.839682:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829126.839688:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829126.839692:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973083264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829126.839700:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829126.839703:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829126.839704:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829126.839707:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5577 00000100:00080000:3.0:1787829126.839717:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829126.839719:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829126.839723:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829126.839739:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829126/real 1787829126] req@ffff9c1a7895c780 x1874668973082880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829126.839739:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973083392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829126.839762:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829126.839765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829126.839766:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829126.839769:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5577 00000100:00080000:3.0:1787829126.839774:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829126.839775:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973082880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829126.839779:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829126.839786:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829126.839791:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829126.839796:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973083520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829126.839797:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829126.839805:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973082880:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829126.839812:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829126.839816:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829126.839818:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829126.839821:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5577 00000100:00080000:3.0:1787829126.839823:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829126.839825:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829126.839828:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829126.839833:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973083008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829126.839838:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973083648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829126.839860:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829126.839873:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829126/real 1787829126] req@ffff9c1a7895cb40 x1874668973083008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829126.839882:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973083008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829126.839888:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829126.839890:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829126.839892:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973083008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829126.839896:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973083136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829126.839908:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829126.839914:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829126/real 1787829126] req@ffff9c1a7895da40 x1874668973083136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829126.839922:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973083136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829126.839940:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829126.839942:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829126.839944:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973083136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829126.839949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973083264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829126.839964:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829126.839971:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829126/real 1787829126] req@ffff9c1a7895e1c0 x1874668973083264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829126.839979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973083264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829126.839985:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829126.839987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829126.839989:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973083264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829126.839992:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973083392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829126.840002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829126.840007:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829126/real 1787829126] req@ffff9c1a7895fc00 x1874668973083392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829126.840019:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973083392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829126.840024:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829126.840026:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829126.840028:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973083392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829126.840031:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973083520:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829126.840041:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829126.840047:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829126/real 1787829126] req@ffff9c1a7895f840 x1874668973083520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829126.840054:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973083520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829126.840059:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829126.840061:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829126.840063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973083520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829126.840076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973083648:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829126.840086:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829126.840092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829126/real 1787829126] req@ffff9c1a7895f0c0 x1874668973083648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829126.840100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973083648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829126.840109:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829126.840111:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829126.840113:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973083648:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829127.351455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829127.351455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829127.607517:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829128.375418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829128.375420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829128.631497:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829129.399457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829129.399464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829129.655522:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829130.423431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829130.423450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829130.679491:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829131.447422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829131.447430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829131.703484:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829131.959464:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829131.959473:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829131.959476:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829131.959484:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5582 00000100:00080000:0.0:1787829131.959494:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829131.959500:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829131.959535:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973083776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829131.959558:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829131.959561:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829131.959562:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829131.959565:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5582 00000100:00080000:0.0:1787829131.959567:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829131.959570:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829131.959578:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829131.959587:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973083904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829131.959595:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829131.959597:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829131.959599:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829131.959601:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5582 00000100:00080000:0.0:1787829131.959603:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829131.959605:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829131.959609:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829131.959613:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973083776:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829131.959618:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973084032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829131.959628:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829131.959630:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829131.959631:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829131.959634:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5582 00000100:00080000:0.0:1787829131.959637:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829131.959639:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829131.959642:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829131.959651:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973084160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829131.959656:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829131.959659:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829131.959660:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829131.959662:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5582 00000100:00080000:0.0:1787829131.959664:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829131.959667:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829131.959671:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829131.959677:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973084288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829131.959685:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829131.959688:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829131.959692:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829131.959693:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829131.959697:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5582 00000100:00080000:0.0:1787829131.959700:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829131.959702:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829131.959706:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829131.959713:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973084416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829131.959719:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829131.959721:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829131.959723:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829131.959726:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5582 00000100:00080000:0.0:1787829131.959728:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829131.959730:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829131.959733:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829131.959739:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829131/real 1787829131] req@ffff9c1a78860000 x1874668973083776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829131.959742:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973084544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829131.959801:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973083776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829131.959815:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829131.959820:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829131.959824:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973083776:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829131.959831:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973083904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829131.959870:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829131.959881:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829131/real 1787829131] req@ffff9c1a78863c00 x1874668973083904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829131.960179:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973083904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829131.960186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829131.960188:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829131.960191:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973083904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829131.960221:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973084032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829131.960264:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829131.960279:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829131/real 1787829131] req@ffff9c1a788630c0 x1874668973084032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829131.960287:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973084032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829131.960293:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829131.960295:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829131.960297:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973084032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829131.960300:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973084160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829131.960313:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829131.960319:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829131/real 1787829131] req@ffff9c1a78861680 x1874668973084160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829131.960326:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973084160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829131.960336:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829131.960338:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829131.960340:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973084160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829131.960343:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973084288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829131.960379:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829131.960386:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829131/real 1787829131] req@ffff9c1a78862580 x1874668973084288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829131.960393:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973084288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829131.960398:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829131.960401:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829131.960403:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973084288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829131.960406:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973084416:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829131.960417:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829131.960423:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829131/real 1787829131] req@ffff9c1a78861e00 x1874668973084416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829131.960430:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973084416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829131.960435:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829131.960437:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829131.960439:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973084416:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829131.960442:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973084544:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829131.960451:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829131.960456:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829131/real 1787829131] req@ffff9c1a78860780 x1874668973084544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829131.960463:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973084544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829131.960468:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829131.960469:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829131.960471:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973084544:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829132.471446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829132.471446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829132.727522:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829133.495440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829133.495440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829133.751526:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829134.519445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829134.519445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829134.775481:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829135.543426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829135.543426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829135.799506:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829136.567417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829136.567417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829136.823506:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829137.079432:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829137.079440:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829137.079443:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829137.079452:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5587 00000100:00080000:3.0:1787829137.079461:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829137.079467:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829137.079495:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973084672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829137.079520:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829137.079524:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829137.079527:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829137.079532:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5587 00000100:00080000:3.0:1787829137.079537:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829137.079541:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829137.079551:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829137.079560:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973084672:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829137.079567:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973084800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829137.079578:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829137.079581:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829137.079583:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829137.079587:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5587 00000100:00080000:3.0:1787829137.079590:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829137.079594:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829137.079598:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829137.079609:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973084928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829137.079618:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829137.079618:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829137.079622:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829137.079623:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829137.079627:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5587 00000100:00080000:3.0:1787829137.079631:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829137.079635:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829137.079641:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829137.079661:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff9c1a7895de00 x1874668973084672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829137.079680:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973084672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829137.079694:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829137.079700:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829137.079704:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973084672:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829137.079730:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973084800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829137.079760:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829137.079777:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff9c1a7895f0c0 x1874668973084800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829137.079788:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973084800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829137.079796:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829137.079800:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829137.079803:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973084800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829137.079810:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973084928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829137.079831:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829137.079842:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff9c1a7895f840 x1874668973084928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829137.079845:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973085056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829137.079871:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973084928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829137.079871:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829137.079875:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829137.079878:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829137.079881:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829137.079884:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829137.079888:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973084928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829137.079911:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973085056:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829137.079924:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5587 00000100:00080000:3.0:1787829137.079928:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829137.079933:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787829137.079934:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829137.079938:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829137.079946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff9c1a7895fc00 x1874668973085056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829137.079949:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973085184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829137.079959:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973085056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829137.079960:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829137.079963:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829137.079965:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829137.079968:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829137.079969:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5587 00000100:00080000:0.0:1787829137.079971:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829137.079972:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829137.079974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973085056:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829137.079976:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829137.079981:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829137.079985:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973085184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829137.079990:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973085312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829137.080000:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829137.080002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829137.080003:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829137.080006:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829137.080009:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5587 00000100:00100000:0.0:1787829137.080012:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff9c1a7895e1c0 x1874668973085184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829137.080012:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829137.080016:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829137.080021:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829137.080023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973085184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829137.080030:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973085440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829137.080032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829137.080036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829137.080039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973085184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829137.080050:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973085312:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829137.080071:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829137.080081:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff9c1a7895da40 x1874668973085312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829137.081141:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973085312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829137.081146:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829137.081148:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829137.081150:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973085312:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829137.081154:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973085440:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829137.081169:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829137.081177:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829137/real 1787829137] req@ffff9c1a7895cb40 x1874668973085440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829137.081183:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973085440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829137.081188:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829137.081190:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829137.081192:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973085440:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829137.591418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829137.591425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829137.847499:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829138.615446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829138.615446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829138.871468:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829139.639416:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829139.639434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829139.895503:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829140.663459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829140.663463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829140.919510:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829141.687436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829141.687445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829141.943488:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829142.199452:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829142.199461:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829142.199464:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829142.199472:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5592 00000100:00080000:3.0:1787829142.199481:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829142.199486:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829142.199516:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973085568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829142.199543:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829142.199546:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829142.199548:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829142.199551:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5592 00000100:00080000:3.0:1787829142.199554:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829142.199557:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829142.199565:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829142.199574:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973085696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829142.199582:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973085568:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829142.199587:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829142.199591:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829142.199592:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829142.199596:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5592 00000100:00080000:3.0:1787829142.199598:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829142.199600:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829142.199604:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829142.199612:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973085824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829142.199621:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829142.199623:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829142.199625:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829142.199626:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5592 00000100:00080000:3.0:1787829142.199632:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829142.199634:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829142.199637:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829142.199642:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829142.199646:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973085952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829142.199655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829142.199657:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829142.199659:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829142.199661:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5592 00000100:00080000:3.0:1787829142.199664:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829142.199667:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829142.199671:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829142.199677:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff9c1a7895c780 x1874668973085568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829142.199684:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973086080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829142.199697:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973085568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829142.199699:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829142.199704:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829142.199707:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829142.199719:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5592 00000100:00080000:0.0:1787829142.199721:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829142.199725:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829142.199730:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829142.199735:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829142.199741:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973085568:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787829142.199742:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829142.199757:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973086208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829142.199764:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829142.199766:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829142.199768:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829142.199773:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5592 00000100:00100000:0.0:1787829142.199776:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973085696:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829142.199777:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829142.199785:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829142.199789:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829142.199798:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973086336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829142.199830:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829142.199848:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff9c1a7895e940 x1874668973085696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829142.199860:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973085696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829142.199866:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829142.199868:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829142.199871:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973085696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829142.199875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973085824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829142.199890:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829142.199897:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff9c1a7895c3c0 x1874668973085824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829142.199905:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973085824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829142.199914:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829142.199916:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829142.199918:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973085824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829142.199922:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973085952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829142.199932:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829142.199938:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff9c1a7895cf00 x1874668973085952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829142.199945:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973085952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829142.199951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829142.199953:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829142.199955:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973085952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829142.199958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973086080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829142.199968:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829142.199973:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff9c1a7895f480 x1874668973086080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829142.199980:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973086080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829142.199985:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829142.199987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829142.199989:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973086080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829142.199993:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973086208:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829142.200003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829142.200008:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff9c1a7895d2c0 x1874668973086208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829142.200015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973086208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829142.200020:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829142.200021:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829142.200023:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973086208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829142.200044:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973086336:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829142.200055:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829142.200062:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829142/real 1787829142] req@ffff9c1a7895c000 x1874668973086336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829142.200069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973086336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829142.200075:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829142.200077:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829142.200079:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973086336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829142.711445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829142.711448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829142.967549:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829143.735440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829143.735440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829143.991487:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829144.759438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829144.759438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829145.015526:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829145.783405:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829145.783421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829146.039506:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829146.807448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829146.807448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829147.063504:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829147.319524:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829147.319534:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829147.319538:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829147.319549:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5597 00000100:00080000:2.0:1787829147.319559:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829147.319566:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829147.319601:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973086464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829147.319631:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829147.319634:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829147.319636:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829147.319640:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5597 00000100:00080000:2.0:1787829147.319643:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829147.319647:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829147.319656:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829147.319667:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973086592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829147.319678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829147.319681:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829147.319686:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829147.319687:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973086464:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829147.319693:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5597 00000100:00080000:2.0:1787829147.319698:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829147.319702:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829147.319708:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829147.319720:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973086720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829147.319730:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829147.319734:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829147.319736:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829147.319740:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5597 00000100:00080000:2.0:1787829147.319743:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829147.319747:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829147.319751:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829147.319757:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829147.319781:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973086848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829147.319803:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829147.319805:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff9c1a49d8cb40 x1874668973086464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829147.319811:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829147.319815:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829147.319822:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5597 00000100:00080000:2.0:1787829147.319826:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829147.319827:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973086464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829147.319835:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787829147.319841:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829147.319843:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829147.319846:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829147.319851:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973086464:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829147.319860:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973086976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829147.319872:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829147.319874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973086592:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829147.319879:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829147.319883:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829147.319888:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5597 00000100:00080000:2.0:1787829147.319892:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829147.319896:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829147.319901:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829147.319906:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829147.319915:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973087104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829147.319926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829147.319929:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829147.319931:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829147.319936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff9c1a49d8cf00 x1874668973086592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829147.319938:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5597 00000100:00080000:2.0:1787829147.319945:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829147.319952:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829147.319956:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973086592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829147.319959:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829147.319968:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829147.319971:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829147.319974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973086592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829147.319979:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973087232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829147.319983:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973086720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829147.320022:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829147.320035:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff9c1a49d8c000 x1874668973086720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829147.320044:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973086720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829147.320057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829147.320059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829147.320062:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973086720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829147.320066:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973086848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829147.320081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829147.320088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff9c1a49d8c780 x1874668973086848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829147.320099:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973086848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829147.320108:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829147.320111:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829147.320114:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973086848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829147.320132:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973086976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829147.320151:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829147.320159:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff9c1a49d8de00 x1874668973086976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829147.320172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973086976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829147.320178:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829147.320180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829147.320182:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973086976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829147.320186:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973087104:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829147.320197:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829147.320202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff9c1a49d8da40 x1874668973087104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829147.320217:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973087104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829147.320223:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829147.320225:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829147.320227:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973087104:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829147.320230:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973087232:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829147.320240:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829147.320246:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829147/real 1787829147] req@ffff9c1a49d8cb40 x1874668973087232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829147.320253:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973087232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829147.320263:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829147.320264:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829147.320266:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973087232:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829147.831411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829147.831431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829148.087464:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829148.855432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829148.855432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829149.111514:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829149.879415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829149.879418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829150.135518:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829150.903456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829150.903498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829151.159597:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829151.927425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829151.927425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829152.183652:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829152.439482:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829152.439495:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829152.439502:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829152.439516:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5602 00000100:00080000:2.0:1787829152.439533:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829152.439542:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829152.439594:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973087360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829152.439624:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829152.439629:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829152.439631:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829152.439636:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5602 00000100:00080000:2.0:1787829152.439640:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829152.439645:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829152.439658:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829152.439668:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973087360:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829152.439675:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973087488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829152.439689:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829152.439692:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829152.439695:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829152.439699:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5602 00000100:00080000:2.0:1787829152.439704:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829152.439708:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829152.439714:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829152.439726:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973087616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829152.439738:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829152.439743:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829152.439745:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829152.439763:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5602 00000100:00080000:2.0:1787829152.439768:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829152.439773:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787829152.439779:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787829152.439782:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829152.439798:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973087744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829152.439809:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829152.439813:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829152.439814:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829152.439819:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5602 00000100:00080000:2.0:1787829152.439823:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829152.439828:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829152.439834:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829152.439839:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff9c1a49d8f0c0 x1874668973087360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829152.439851:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973087872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829152.439864:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829152.439870:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973087360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829152.439870:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829152.439875:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829152.439886:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5602 00000100:00080000:2.0:1787829152.439892:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829152.439893:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829152.439899:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787829152.439904:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:2.0:1787829152.439910:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829152.439912:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973087360:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829152.439929:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973088000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829152.439941:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829152.439945:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829152.439947:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829152.439950:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973087488:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829152.439955:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5602 00000100:00080000:2.0:1787829152.439962:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829152.439967:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829152.439972:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829152.439986:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973088128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829152.439990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829152.440020:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff9c1a49d8d680 x1874668973087488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829152.440037:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973087488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829152.440047:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829152.440051:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829152.440054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973087488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829152.440061:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973087616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829152.440082:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829152.440092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff9c1a49d8ed00 x1874668973087616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829152.440113:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973087616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829152.440121:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829152.440125:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829152.440128:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973087616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829152.440135:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973087744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829152.440156:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829152.440166:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff9c1a49d8fc00 x1874668973087744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829152.440178:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973087744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829152.440187:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829152.440191:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829152.440195:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973087744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829152.440201:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973087872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829152.440219:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829152.440229:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff9c1a49d8e580 x1874668973087872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829152.440241:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973087872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829152.440261:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829152.440264:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829152.440267:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973087872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829152.440274:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973088000:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829152.440294:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829152.440305:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff9c1a49d8c3c0 x1874668973088000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829152.440316:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973088000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829152.440324:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829152.440328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829152.440331:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973088000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829152.440378:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973088128:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829152.440402:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829152.440416:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829152/real 1787829152] req@ffff9c1a49d8e1c0 x1874668973088128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829152.440430:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973088128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829152.440439:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829152.440442:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829152.440446:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973088128:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829152.951501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829152.951502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829153.207515:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829153.975457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829153.975457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829154.231508:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829154.999443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829154.999443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829155.255499:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829156.023471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829156.023471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829156.279565:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829157.047441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829157.047441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829157.303584:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829157.559442:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829157.559453:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829157.559457:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829157.559467:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5607 00000100:00080000:2.0:1787829157.559479:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829157.559486:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829157.559528:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973088256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829157.559558:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829157.559562:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829157.559564:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829157.559568:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5607 00000100:00080000:2.0:1787829157.559571:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829157.559574:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829157.559584:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829157.559594:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973088384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829157.559604:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829157.559607:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829157.559609:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829157.559612:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5607 00000100:00080000:2.0:1787829157.559615:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829157.559619:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829157.559623:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829157.559632:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973088512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829157.559643:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829157.559646:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829157.559649:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829157.559652:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5607 00000100:00080000:2.0:1787829157.559659:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829157.559660:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973088256:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829157.559667:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829157.559676:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829157.559688:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973088640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829157.559698:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829157.559702:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829157.559704:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829157.559707:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5607 00000100:00080000:2.0:1787829157.559732:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829157.559739:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829157.559745:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829157.559760:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973088768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829157.559769:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829157.559778:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829157.559783:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829157.559785:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829157.559790:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5607 00000100:00080000:2.0:1787829157.559806:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829157.559813:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829157.559821:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829157.559839:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973088896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829157.559844:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829157/real 1787829157] req@ffff9c1a49d8e1c0 x1874668973088256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829157.559888:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829157.559894:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829157.559895:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973088256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829157.559900:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829157.559913:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5607 00000100:00080000:0.0:1787829157.559920:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829157.559921:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829157.559931:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787829157.559933:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829157.559942:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973088256:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787829157.559943:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829157.559959:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973088384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829157.559969:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973089024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829157.559995:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829157.560013:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829157/real 1787829157] req@ffff9c1a49d8c3c0 x1874668973088384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829157.560026:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973088384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829157.560034:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829157.560037:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829157.560040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973088384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829157.560046:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973088512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829157.560061:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829157.560070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829157/real 1787829157] req@ffff9c1a49d8e580 x1874668973088512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829157.560080:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973088512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829157.560088:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829157.560090:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829157.560093:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973088512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829157.560137:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973088640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829157.560166:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829157.560183:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829157/real 1787829157] req@ffff9c1a49d8fc00 x1874668973088640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829157.560202:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973088640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829157.560214:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829157.560217:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829157.560221:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973088640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829157.560227:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973088768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829157.560248:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829157.560259:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829157/real 1787829157] req@ffff9c1a49d8ed00 x1874668973088768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829157.560270:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973088768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829157.560277:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829157.560280:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829157.560283:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973088768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829157.560288:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973088896:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829157.560302:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829157.560310:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829157/real 1787829157] req@ffff9c1a49d8d680 x1874668973088896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829157.560321:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973088896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829157.560328:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829157.560330:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829157.560334:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973088896:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829157.560339:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973089024:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829157.560391:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829157.560403:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829157/real 1787829157] req@ffff9c1a49d8f0c0 x1874668973089024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829157.560413:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973089024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829157.560421:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829157.560424:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829157.560427:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973089024:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829158.071448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829158.071449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829158.327680:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829159.095419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829159.095437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829159.351541:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829160.119453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829160.119453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829160.375630:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829161.143417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829161.143421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829161.399450:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829162.167456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829162.167459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829162.423500:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829162.679510:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829162.679520:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829162.679525:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829162.679535:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5613 00000100:00080000:3.0:1787829162.679546:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829162.679553:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829162.679610:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973089152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829162.679644:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829162.679648:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829162.679650:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829162.679655:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5613 00000100:00080000:3.0:1787829162.679659:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829162.679663:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829162.679673:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829162.679725:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973089280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829162.679736:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829162.679739:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829162.679741:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829162.679746:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5613 00000100:00080000:3.0:1787829162.679750:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829162.679754:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829162.679760:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829162.679764:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973089152:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829162.679773:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973089408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829162.679785:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829162.679789:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829162.679791:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829162.679794:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5613 00000100:00080000:3.0:1787829162.679797:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829162.679800:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829162.679804:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829162.679816:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973089536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829162.679825:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829162.679828:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829162.679830:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829162.679832:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5613 00000100:00080000:3.0:1787829162.679836:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829162.679843:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829162.679846:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829162.679852:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829162.679863:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973089664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829162.679872:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829162.679875:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829162.679877:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829162.679882:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5613 00000100:00080000:3.0:1787829162.679886:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829162.679891:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829162/real 1787829162] req@ffff9c1a7895e580 x1874668973089152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829162.679895:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829162.679904:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829162.679923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973089152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829162.679924:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973089792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829162.679964:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829162.679967:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829162.679971:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829162.679980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829162.679983:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973089152:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829162.679987:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829162.679994:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973089280:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829162.679999:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5613 00000100:00080000:3.0:1787829162.680009:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829162.680017:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787829162.680025:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829162.680026:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829162.680040:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829162/real 1787829162] req@ffff9c1a7895c000 x1874668973089280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829162.680047:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973089920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829162.680070:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973089280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829162.680080:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829162.680084:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829162.680087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973089280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829162.680128:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973089408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829162.680146:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829162.680156:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829162/real 1787829162] req@ffff9c1a7895d2c0 x1874668973089408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829162.680166:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973089408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829162.680174:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829162.680176:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829162.680179:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973089408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829162.680184:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973089536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829162.680197:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829162.680205:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829162/real 1787829162] req@ffff9c1a7895f480 x1874668973089536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829162.680215:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973089536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829162.680222:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829162.680224:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829162.680226:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973089536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829162.680231:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973089664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829162.680244:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829162.680252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829162/real 1787829162] req@ffff9c1a7895cf00 x1874668973089664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829162.680262:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973089664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829162.680269:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829162.680271:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829162.680274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973089664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829162.680278:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973089792:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829162.680298:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829162.680312:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829162/real 1787829162] req@ffff9c1a7895c3c0 x1874668973089792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829162.680330:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973089792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829162.680343:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829162.680384:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829162.680388:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973089792:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829162.680394:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973089920:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829162.680417:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829162.680428:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829162/real 1787829162] req@ffff9c1a7895e940 x1874668973089920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829162.680439:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973089920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829162.680446:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829162.680449:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829162.680452:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973089920:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829163.191443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829163.191443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829163.447464:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829164.215442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829164.215442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829164.471549:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829165.239411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829165.239411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829165.495573:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829166.263447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829166.263447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829166.519637:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829167.287440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829167.287446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829167.543524:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829167.799440:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829167.799449:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829167.799453:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829167.799462:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5618 00000100:00080000:0.0:1787829167.799471:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829167.799477:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829167.799516:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973090048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829167.799538:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829167.799541:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829167.799542:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829167.799545:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5618 00000100:00080000:0.0:1787829167.799548:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829167.799550:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829167.799557:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829167.799566:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973090176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829167.799574:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829167.799576:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829167.799578:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829167.799580:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5618 00000100:00080000:0.0:1787829167.799583:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829167.799588:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973090048:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829167.799589:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829167.799596:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829167.799603:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973090304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829167.799611:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829167.799614:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829167.799615:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829167.799618:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5618 00000100:00080000:0.0:1787829167.799621:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829167.799624:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829167.799627:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829167.799635:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973090432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829167.799653:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829167.799656:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829167.799657:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829167.799660:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5618 00000100:00080000:0.0:1787829167.799663:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829167.799665:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829167.799668:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829167.799676:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973090560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829167.799683:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829167.799685:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829167.799687:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829167.799690:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5618 00000100:00080000:0.0:1787829167.799693:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829167.799699:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829167.799699:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973090176:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787829167.799708:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829167.799720:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973090688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829167.799728:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829167.799731:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829167.799732:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829167.799736:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5618 00000100:00080000:0.0:1787829167.799738:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829167.799740:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829167.799744:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829167.799746:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973090304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829167.799755:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973090816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829167.799775:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973090432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829167.799790:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973090560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829167.799801:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973090688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829167.799816:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973090816:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829168.311455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829168.311455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829168.567537:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829169.335442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829169.335460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829169.591523:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829170.359498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829170.359498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829170.615547:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829170.871790:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:3.0:1787829170.871847:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829170.871884:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829170.871943:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829170.871957:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829170.871987:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829170.872001:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829170.872015:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829170.872030:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787829170.872033:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829167/real 1787829170] req@ffff9c1a788612c0 x1874668973090304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.872069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973090304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.872096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.872102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829170.872107:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973090304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829170.872115:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829167/real 1787829170] req@ffff9c1a78862940 x1874668973090432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.872133:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973090432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.872167:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.872172:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829170.872177:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973090432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829170.872186:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829167/real 1787829170] req@ffff9c1a78861a40 x1874668973090560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.872201:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973090560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.872212:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.872216:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829170.872221:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973090560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829170.872230:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829167/real 1787829170] req@ffff9c1a78860f00 x1874668973090688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.872247:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973090688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.872259:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.872264:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829170.872268:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973090688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829170.872278:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829167/real 1787829170] req@ffff9c1a788621c0 x1874668973090048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.872295:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973090048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.872307:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.872312:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829170.872317:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973090048:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829170.872326:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829167/real 1787829170] req@ffff9c1a78863840 x1874668973090176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.872341:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973090176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.872385:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.872390:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829170.872395:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973090176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829170.872438:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829167/real 1787829170] req@ffff9c1a78862d00 x1874668973090816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829170.872458:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973090816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829170.872468:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829170.872473:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829170.872477:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973090816:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829171.383396:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829171.383433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829171.639488:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829172.407487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829172.407491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829172.663638:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829172.919514:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829172.919530:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829172.919536:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829172.919550:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5623 00000100:00080000:0.0:1787829172.919567:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829172.919577:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829172.919637:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973090944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829172.919677:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829172.919682:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829172.919686:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829172.919692:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5623 00000100:00080000:0.0:1787829172.919698:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829172.919705:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829172.919719:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829172.919727:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973090944:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829172.919740:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973091072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829172.919753:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829172.919757:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829172.919760:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829172.919765:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5623 00000100:00080000:0.0:1787829172.919769:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829172.919774:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829172.919780:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829172.919796:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973091200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829172.919809:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829172.919814:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829172.919816:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829172.919822:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5623 00000100:00080000:0.0:1787829172.919826:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829172.919833:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829172.919840:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829172.919854:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973091328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829172.919864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829172.919868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829172.919870:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829172.919875:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5623 00000100:00080000:0.0:1787829172.919879:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829172.919885:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829172.919891:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829172.919904:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973091456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829172.919923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829172.919928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829172.919930:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829172.919939:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5623 00000800:00000100:3.0:1787829172.919946:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829172.919948:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829172.919960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829172.919969:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829172.919987:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973091584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829172.920010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829172.920011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829172/real 1787829172] req@ffff9c1a78860b40 x1874668973090944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829172.920025:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829172.920029:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829172.920041:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5623 00000100:00080000:0.0:1787829172.920048:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829172.920055:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829172.920061:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973090944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829172.920068:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829172.920089:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829172.920093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973091712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829172.920101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829172.920121:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973090944:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829172.920166:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973091072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829172.920201:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829172.920218:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829172/real 1787829172] req@ffff9c1a78862d00 x1874668973091072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829172.920232:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973091072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829172.920242:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829172.920246:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829172.920250:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973091072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829172.920258:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973091200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829172.920281:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829172.920294:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829172/real 1787829172] req@ffff9c1a78863840 x1874668973091200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829172.920306:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973091200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829172.920314:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829172.920318:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829172.920321:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973091200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829172.920328:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973091328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829172.920619:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829172.920661:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829172/real 1787829172] req@ffff9c1a788621c0 x1874668973091328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829172.920679:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973091328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829172.920689:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829172.920693:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829172.920712:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973091328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829172.920721:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973091456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829172.920746:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829172.920793:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829172/real 1787829172] req@ffff9c1a78860f00 x1874668973091456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829172.920807:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973091456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829172.920816:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829172.920820:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829172.920824:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973091456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829172.920832:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973091584:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829172.920851:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829172.920861:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829172/real 1787829172] req@ffff9c1a78861a40 x1874668973091584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829172.920874:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973091584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829172.920882:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829172.920886:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829172.920889:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973091584:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829172.920896:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973091712:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829172.920915:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829172.920927:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829172/real 1787829172] req@ffff9c1a78862940 x1874668973091712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829172.920938:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973091712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829172.920947:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829172.920950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829172.920954:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973091712:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829173.431465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829173.431468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829173.687523:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829174.455434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829174.455435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829174.711574:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829175.479443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829175.479443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829175.735474:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829176.503418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829176.503439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829176.759540:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829177.527471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829177.527471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829177.783531:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829178.039476:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829178.039505:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829178.039510:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.039521:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5628 00000100:00080000:2.0:1787829178.039537:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.039545:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829178.039602:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973091840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829178.039645:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829178.039648:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829178.039650:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.039654:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5628 00000100:00080000:2.0:1787829178.039657:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.039661:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829178.039672:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829178.039681:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973091840:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829178.039685:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973091968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829178.039696:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829178.039698:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829178.039699:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.039702:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5628 00000100:00080000:2.0:1787829178.039704:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.039706:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829178.039710:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829178.039717:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973092096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829178.039727:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829178.039730:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829178.039731:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.039733:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5628 00000100:00080000:2.0:1787829178.039754:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.039756:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829178.039759:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829178.039768:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829178.039769:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973092224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829178.039778:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829178.039781:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829178.039783:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.039786:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5628 00000100:00080000:2.0:1787829178.039789:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.039792:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829178.039795:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829178.039803:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973092352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829178.039810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829178.039812:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829178.039813:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829178.039819:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5628 00000100:00100000:3.0:1787829178.039820:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff9c1a49d8cb40 x1874668973091840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829178.039823:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.039830:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829178.039838:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829178.039847:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973091840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829178.039849:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973092480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829178.039872:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829178.039873:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829178.039878:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829178.039883:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829178.039883:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829178.039891:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5628 00000100:00100000:3.0:1787829178.039891:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973091840:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829178.039898:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829178.039902:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829178.039905:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829178.039914:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973092608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829178.039924:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973091968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829178.039953:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829178.039968:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff9c1a49d8da40 x1874668973091968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829178.039981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973091968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.039988:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.039990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829178.039993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973091968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829178.039998:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973092096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829178.040014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829178.040022:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff9c1a49d8de00 x1874668973092096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829178.040029:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973092096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.040035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.040037:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829178.040038:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973092096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829178.040042:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973092224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829178.040053:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829178.040059:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff9c1a49d8c780 x1874668973092224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829178.040067:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973092224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.040073:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.040074:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829178.040076:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973092224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829178.040088:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973092352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829178.040103:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829178.040111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff9c1a49d8c000 x1874668973092352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829178.040118:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973092352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.040124:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.040126:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829178.040129:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973092352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829178.040132:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973092480:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829178.040144:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829178.040150:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff9c1a49d8cf00 x1874668973092480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829178.040157:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973092480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.040163:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.040164:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829178.040166:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973092480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829178.040181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973092608:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829178.040192:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829178.040198:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829178/real 1787829178] req@ffff9c1a49d8f0c0 x1874668973092608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829178.040205:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973092608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829178.040211:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829178.040213:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829178.040215:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973092608:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829178.551410:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829178.551422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829178.807497:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829179.575428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829179.575434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829179.831586:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829180.599438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829180.599474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829180.855530:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829181.623478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829181.623478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829181.879503:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829182.647449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829182.647449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829182.903531:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829183.159494:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829183.159501:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829183.159505:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829183.159513:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5633 00000100:00080000:3.0:1787829183.159522:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829183.159528:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829183.159595:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973092736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829183.159626:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829183.159629:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829183.159631:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829183.159634:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5633 00000100:00080000:3.0:1787829183.159637:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829183.159641:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829183.159650:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829183.159676:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973092864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829183.159685:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829183.159687:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829183.159691:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829183.159692:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973092736:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829183.159696:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5633 00000100:00080000:3.0:1787829183.159702:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829183.159706:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829183.159709:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829183.159717:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973092992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829183.159724:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829183.159743:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829183.159745:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829183.159748:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5633 00000100:00080000:3.0:1787829183.159760:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829183.159763:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829183.159767:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829183.159771:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829183.159779:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973093120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829183.159788:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829183.159790:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829183.159792:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829183.159794:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5633 00000100:00080000:3.0:1787829183.159796:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829183.159800:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829183.159806:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829183.159808:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff9c1a7895c780 x1874668973092736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829183.159822:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973093248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829183.159834:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973092736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829183.159837:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829183.159849:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829183.159852:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829183.159854:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829183.159859:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829183.159863:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5633 00000100:00100000:2.0:1787829183.159867:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973092736:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829183.159869:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829183.159876:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829183.159880:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829183.159889:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973092864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829183.159893:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973093376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829183.159903:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829183.159906:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829183.159907:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829183.159913:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5633 00000800:00000100:2.0:1787829183.159914:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829183.159919:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829183.159924:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829183.159928:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829183.159929:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff9c1a7895cb40 x1874668973092864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829183.159939:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973093504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829183.159944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973092864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829183.159960:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829183.159963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829183.159966:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973092864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829183.159972:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973092992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829183.159986:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829183.159992:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff9c1a7895da40 x1874668973092992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829183.160000:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973092992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829183.160010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829183.160012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829183.160014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973092992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829183.160018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973093120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829183.160028:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829183.160034:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff9c1a7895e1c0 x1874668973093120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829183.160041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973093120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829183.160046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829183.160048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829183.160050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973093120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829183.160054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973093248:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829183.160064:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829183.160070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff9c1a7895fc00 x1874668973093248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829183.160077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973093248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829183.160082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829183.160084:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829183.160087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973093248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829183.160098:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973093376:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829183.160125:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829183.160132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff9c1a7895f840 x1874668973093376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829183.160138:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973093376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829183.160144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829183.160146:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829183.160148:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973093376:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829183.160151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973093504:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829183.160160:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829183.160165:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829183/real 1787829183] req@ffff9c1a7895f0c0 x1874668973093504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829183.160172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973093504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829183.160177:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829183.160179:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829183.160181:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973093504:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829183.671450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829183.671450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829183.927576:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829184.695451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829184.695456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829184.951509:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829185.719425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829185.719450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829185.975571:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829186.743455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829186.743486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829186.999568:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829187.767517:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829187.767520:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829188.023596:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829188.279525:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829188.279532:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829188.279535:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829188.279544:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5638 00000100:00080000:2.0:1787829188.279553:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829188.279558:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829188.279591:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973093632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829188.279619:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829188.279621:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829188.279623:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829188.279626:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5638 00000100:00080000:2.0:1787829188.279628:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829188.279631:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829188.279639:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829188.279648:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973093760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829188.279655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829188.279657:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829188.279658:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829188.279660:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5638 00000100:00080000:2.0:1787829188.279662:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829188.279664:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829188.279668:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829188.279675:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973093888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829188.279684:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829188.279689:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829188.279692:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829188.279692:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973093632:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829188.279708:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5638 00000100:00080000:2.0:1787829188.279714:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829188.279718:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829188.279723:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829188.279741:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973094016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829188.279751:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829188.279754:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829188.279755:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829188.279758:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5638 00000100:00080000:2.0:1787829188.279779:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829188.279782:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829188.279786:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829188.279793:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829188.279799:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973094144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829188.279807:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829188.279809:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829188.279811:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829188.279815:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5638 00000100:00080000:2.0:1787829188.279836:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829188.279839:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829188.279844:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829188.279854:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973094272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829188.279861:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829188.279864:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829188.279870:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829188.279871:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff9c1a49d8d680 x1874668973093632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829188.279876:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5638 00000100:00080000:2.0:1787829188.279882:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829188.279889:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829188.279900:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829188.279930:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973094400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829188.279936:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973093632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.279955:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.279962:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829188.279970:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973093632:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829188.279982:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973093760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829188.280008:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829188.280021:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff9c1a49d8ed00 x1874668973093760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829188.280045:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973093760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.280053:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.280056:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829188.280059:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973093760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829188.280090:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973093888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829188.280124:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829188.280136:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff9c1a49d8fc00 x1874668973093888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829188.280148:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973093888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.280161:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.280165:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829188.280168:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973093888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829188.280176:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973094016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829188.280195:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829188.280205:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff9c1a49d8e580 x1874668973094016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829188.280226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973094016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.280234:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.280238:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829188.280242:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973094016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829188.280248:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973094144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829188.280269:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829188.280291:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff9c1a49d8c3c0 x1874668973094144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829188.280304:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973094144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.280325:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.280328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829188.280331:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973094144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829188.280336:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973094272:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829188.280376:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829188.280388:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff9c1a49d8e1c0 x1874668973094272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829188.280401:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973094272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.280410:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.280413:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829188.280417:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973094272:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829188.280424:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973094400:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829188.280445:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829188.280458:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829188/real 1787829188] req@ffff9c1a49d8f0c0 x1874668973094400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829188.280470:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973094400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829188.280479:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829188.280483:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829188.280486:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973094400:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829188.791458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829188.791469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829189.047481:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829189.815466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829189.815466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829190.071592:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829190.839450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829190.839452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829191.095514:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829191.863457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829191.863458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829192.119579:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829192.887509:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829192.887509:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829193.143577:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829193.399554:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829193.399562:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829193.399566:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829193.399586:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5643 00000100:00080000:3.0:1787829193.399595:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829193.399601:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829193.399633:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973094528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829193.399663:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829193.399667:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829193.399669:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829193.399675:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5643 00000100:00080000:3.0:1787829193.399680:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829193.399685:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829193.399695:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829193.399728:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973094656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829193.399745:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973094528:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829193.399748:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829193.399767:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829193.399781:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829193.399786:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5643 00000100:00080000:3.0:1787829193.399792:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829193.399814:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829193.399821:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829193.399837:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973094784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829193.399848:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829193.399851:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829193.399853:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829193.399865:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5643 00000100:00080000:3.0:1787829193.399870:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829193.399871:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829193.399879:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829193.399885:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829193.399902:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973094912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829193.399913:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829193.399917:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829193.399920:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829193.399928:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5643 00000100:00100000:2.0:1787829193.399929:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff9c1a7895de00 x1874668973094528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829193.399942:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829193.399954:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829193.399961:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973094528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829193.399964:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829193.399983:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829193.399989:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973095040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829193.399992:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829193.400010:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973094528:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829193.400025:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829193.400030:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829193.400033:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829193.400038:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5643 00000100:00080000:3.0:1787829193.400041:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829193.400045:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829193.400052:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829193.400063:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973095168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829193.400067:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973094656:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829193.400095:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829193.400100:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829193.400103:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829193.400107:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5643 00000100:00080000:3.0:1787829193.400113:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829193.400120:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829193.400125:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829193.400134:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829193.400143:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff9c1a7895f0c0 x1874668973094656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829193.400149:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973095296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829193.400180:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973094656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829193.400194:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829193.400199:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829193.400204:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973094656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829193.400212:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973094784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829193.400226:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829193.400232:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff9c1a7895f840 x1874668973094784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829193.400240:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973094784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829193.400245:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829193.400247:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829193.400249:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973094784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829193.400253:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973094912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829193.400264:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829193.400270:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff9c1a7895fc00 x1874668973094912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829193.400278:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973094912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829193.400283:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829193.400285:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829193.400287:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973094912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829193.400290:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973095040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829193.400300:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829193.400306:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff9c1a7895e1c0 x1874668973095040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829193.400313:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973095040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829193.400318:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829193.400320:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829193.400322:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973095040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829193.400337:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973095168:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829193.400372:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829193.400380:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff9c1a7895da40 x1874668973095168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829193.400393:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973095168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829193.400399:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829193.400401:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829193.400403:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973095168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829193.400407:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973095296:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829193.400418:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829193.400424:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829193/real 1787829193] req@ffff9c1a7895cb40 x1874668973095296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829193.400432:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973095296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829193.400437:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829193.400439:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829193.400441:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973095296:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829193.911448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829193.911450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829194.167507:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829194.935528:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829194.935532:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829195.191554:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829195.959508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829195.959521:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829196.215599:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829196.983446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829196.983447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829197.239691:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829198.007425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829198.007516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829198.263681:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829198.519473:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829198.519485:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829198.519489:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829198.519498:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5648 00000100:00080000:3.0:1787829198.519507:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829198.519513:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829198.519546:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973095424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829198.519570:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829198.519574:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829198.519575:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829198.519579:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5648 00000100:00080000:3.0:1787829198.519581:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829198.519583:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829198.519591:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829198.519599:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973095552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829198.519607:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829198.519616:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829198.519618:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829198.519621:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5648 00000100:00080000:3.0:1787829198.519624:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829198.519626:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829198.519631:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829198.519638:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973095680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829198.519646:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829198.519648:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829198.519650:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829198.519652:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5648 00000100:00080000:3.0:1787829198.519654:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829198.519658:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829198.519662:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829198.519670:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973095808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829198.519684:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973095424:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829198.519685:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829198.519699:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829198.519702:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829198.519706:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5648 00000100:00080000:3.0:1787829198.519708:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829198.519721:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829198.519724:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829198.519730:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973095936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829198.519737:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829198.519741:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829198.519743:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829198.519749:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5648 00000100:00080000:3.0:1787829198.519754:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829198.519759:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829198.519767:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829198.519780:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973096064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829198.519791:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829198.519793:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829198.519796:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829198.519800:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829198.519804:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5648 00000100:00080000:3.0:1787829198.519809:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829198.519812:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829198.519817:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829198.519829:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973096192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829198.519858:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829198/real 1787829198] req@ffff9c1a7895c780 x1874668973095424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829198.519907:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973095424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829198.519932:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829198.519937:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829198.519941:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973095424:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829198.519950:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973095552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829198.519991:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829198.520002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829198/real 1787829198] req@ffff9c1a7895e940 x1874668973095552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829198.520020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973095552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829198.520027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829198.520030:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829198.520032:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973095552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829198.520038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973095680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829198.520052:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829198.520060:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829198/real 1787829198] req@ffff9c1a7895c3c0 x1874668973095680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829198.520068:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973095680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829198.520074:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829198.520076:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829198.520078:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973095680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829198.520110:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973095808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829198.520123:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829198.520131:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829198/real 1787829198] req@ffff9c1a7895cf00 x1874668973095808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829198.520139:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973095808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829198.520146:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829198.520148:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829198.520151:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973095808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829198.520155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973095936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829198.520166:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829198.520172:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829198/real 1787829198] req@ffff9c1a7895f480 x1874668973095936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829198.520191:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973095936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829198.520197:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829198.520199:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829198.520201:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973095936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829198.520205:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973096064:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829198.520225:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829198.520233:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829198/real 1787829198] req@ffff9c1a7895d2c0 x1874668973096064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829198.520241:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973096064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829198.520247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829198.520249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829198.520251:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973096064:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829198.520256:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973096192:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829198.520267:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829198.520274:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829198/real 1787829198] req@ffff9c1a7895c000 x1874668973096192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829198.520282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973096192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829198.520298:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829198.520300:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829198.520302:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973096192:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829199.031466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829199.031466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829199.287730:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829200.055451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829200.055454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829200.311601:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829201.079536:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829201.079536:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829201.335621:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829202.103444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829202.103470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829202.359561:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829203.127450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829203.127450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829203.383593:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829203.639448:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829203.639456:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829203.639460:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829203.639467:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5653 00000100:00080000:2.0:1787829203.639476:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829203.639481:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829203.639513:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973096320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829203.639536:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829203.639539:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829203.639540:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829203.639543:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5653 00000100:00080000:2.0:1787829203.639546:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829203.639549:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829203.639556:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829203.639566:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973096448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829203.639570:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973096320:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829203.639581:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829203.639584:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829203.639586:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829203.639589:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5653 00000100:00080000:2.0:1787829203.639592:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829203.639594:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829203.639597:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829203.639605:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973096576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829203.639613:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829203.639615:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829203.639617:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829203.639620:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5653 00000100:00080000:2.0:1787829203.639629:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829203.639632:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829203.639635:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829203.639639:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829203.639647:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973096704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829203.639656:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829203.639658:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829203.639659:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829203.639662:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5653 00000100:00080000:2.0:1787829203.639664:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829203.639667:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829203.639670:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829203.639676:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829203/real 1787829203] req@ffff9c1a49d8cf00 x1874668973096320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829203.639681:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973096832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829203.639701:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829203.639704:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829203.639705:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973096320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829203.639706:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829203.639715:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5653 00000100:00080000:2.0:1787829203.639720:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829203.639723:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829203.639724:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829203.639731:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829203.639731:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829203.639737:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973096320:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829203.639745:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973096960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829203.639758:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829203.639761:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829203.639763:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829203.639767:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973096448:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829203.639771:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5653 00000100:00080000:2.0:1787829203.639778:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829203.639783:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829203.639789:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829203.639790:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829203.639801:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829203/real 1787829203] req@ffff9c1a49d8c000 x1874668973096448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829203.639806:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973097088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829203.639821:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973096448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829203.639833:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829203.639835:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829203.639839:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973096448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829203.639845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973096576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829203.639856:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829203.639862:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829203/real 1787829203] req@ffff9c1a49d8c780 x1874668973096576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829203.639870:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973096576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829203.639875:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829203.639877:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829203.639879:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973096576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829203.639882:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973096704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829203.639892:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829203.639897:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829203/real 1787829203] req@ffff9c1a49d8de00 x1874668973096704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829203.639904:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973096704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829203.639910:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829203.639911:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829203.639914:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973096704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829203.639917:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973096832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829203.639926:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829203.639931:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829203/real 1787829203] req@ffff9c1a49d8da40 x1874668973096832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829203.639938:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973096832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829203.639942:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829203.639944:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829203.639946:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973096832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829203.639949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973096960:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829203.639959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829203.639964:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829203/real 1787829203] req@ffff9c1a49d8cb40 x1874668973096960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829203.639970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973096960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829203.639975:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829203.639977:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829203.639979:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973096960:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829203.639992:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973097088:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829203.640001:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829203.640007:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829203/real 1787829203] req@ffff9c1a49d8cf00 x1874668973097088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829203.640014:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973097088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829203.640019:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829203.640021:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829203.640023:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973097088:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829204.151439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829204.151461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829204.407501:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829205.175461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829205.175461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829205.431467:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829206.199462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829206.199463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829206.455527:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829207.223450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829207.223459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829207.479547:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829208.247439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829208.247455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829208.503516:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829208.759474:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829208.759483:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829208.759486:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829208.759496:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5659 00000100:00080000:0.0:1787829208.759506:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829208.759513:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829208.759541:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973097216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829208.759563:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829208.759567:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829208.759569:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829208.759573:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5659 00000100:00080000:0.0:1787829208.759576:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829208.759579:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829208.759587:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829208.759599:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973097216:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829208.759600:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973097344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829208.759615:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829208.759617:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829208.759619:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829208.759621:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5659 00000100:00080000:0.0:1787829208.759624:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829208.759626:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829208.759630:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829208.759639:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973097472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829208.759656:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829208.759660:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829208.759663:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829208.759667:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5659 00000100:00080000:0.0:1787829208.759670:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829208.759673:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829208.759679:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829208.759686:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829208.759701:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973097600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829208.759712:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829208.759716:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829208.759718:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829208.759722:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5659 00000100:00080000:0.0:1787829208.759726:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829208.759730:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829208/real 1787829208] req@ffff9c1a788612c0 x1874668973097216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829208.759733:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829208.759740:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829208.759753:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973097728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829208.759755:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973097216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829208.759771:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829208.759776:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829208.759779:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829208.759780:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973097216:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829208.759785:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829208.759789:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829208.759793:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5659 00000100:00080000:0.0:1787829208.759796:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829208.759799:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829208.759802:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829208.759809:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973097344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829208.759813:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973097856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829208.759825:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829208.759828:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829208.759829:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829208.759831:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5659 00000100:00080000:0.0:1787829208.759834:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829208.759837:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829208.759840:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829208.759847:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829208.759850:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973097984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829208.759867:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829208/real 1787829208] req@ffff9c1a78860780 x1874668973097344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829208.759884:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973097344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829208.759894:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829208.759897:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829208.759901:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973097344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829208.759909:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973097472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829208.759929:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829208.759938:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829208/real 1787829208] req@ffff9c1a78861e00 x1874668973097472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829208.759947:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973097472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829208.759953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829208.759955:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829208.759957:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973097472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829208.759962:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973097600:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829208.759976:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829208.759988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829208/real 1787829208] req@ffff9c1a78862580 x1874668973097600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829208.760001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973097600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829208.760018:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829208.760022:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829208.760026:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973097600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829208.760034:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973097728:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829208.760059:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829208.760073:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829208/real 1787829208] req@ffff9c1a78861680 x1874668973097728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829208.760088:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973097728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829208.760102:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829208.760105:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829208.760109:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973097728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829208.760127:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973097856:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829208.760145:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829208.760157:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829208/real 1787829208] req@ffff9c1a788630c0 x1874668973097856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829208.760169:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973097856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829208.760177:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829208.760181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829208.760185:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973097856:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829208.760191:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973097984:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829208.760207:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829208.760214:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829208/real 1787829208] req@ffff9c1a78863c00 x1874668973097984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829208.760221:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973097984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829208.760227:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829208.760229:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829208.760230:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973097984:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829209.271475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829209.271475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829209.527525:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829210.295412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829210.295419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829210.551464:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829211.319435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829211.319435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829211.575541:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829212.343456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829212.343477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829212.599520:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829213.367437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829213.367465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829213.623508:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829213.879410:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829213.879418:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829213.879421:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829213.879430:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5664 00000100:00080000:0.0:1787829213.879440:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829213.879445:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829213.879487:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973098112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829213.879510:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829213.879512:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829213.879514:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829213.879517:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5664 00000100:00080000:0.0:1787829213.879519:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829213.879521:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829213.879539:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829213.879548:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973098240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829213.879555:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829213.879558:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829213.879560:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829213.879563:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5664 00000100:00080000:0.0:1787829213.879565:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829213.879567:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829213.879570:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829213.879577:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973098368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829213.879581:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973098112:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829213.879588:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829213.879591:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829213.879592:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829213.879594:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5664 00000100:00080000:0.0:1787829213.879596:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829213.879598:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829213.879601:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829213.879609:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973098496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829213.879616:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829213.879618:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829213.879619:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829213.879622:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5664 00000100:00080000:0.0:1787829213.879625:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829213.879627:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829213.879629:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829213.879636:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973098624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829213.879643:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829213.879646:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829213.879649:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829213.879652:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829213.879656:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5664 00000100:00080000:0.0:1787829213.879662:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829213.879667:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829213.879673:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829213.879686:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973098752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829213.879692:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829213/real 1787829213] req@ffff9c1a78860000 x1874668973098112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829213.879712:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829213.879717:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829213.879720:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973098112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829213.879721:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829213.879732:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5664 00000100:00080000:2.0:1787829213.879735:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829213.879738:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829213.879741:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829213.879748:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829213.879754:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829213.879756:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973098112:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829213.879765:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973098240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829213.879771:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973098880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829213.879792:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829213.879802:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829213/real 1787829213] req@ffff9c1a78863c00 x1874668973098240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829213.879822:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973098240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829213.879828:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829213.879830:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829213.879832:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973098240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829213.879854:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973098368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829213.879875:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829213.879887:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829213/real 1787829213] req@ffff9c1a788630c0 x1874668973098368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829213.879899:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973098368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829213.879905:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829213.879907:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829213.879910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973098368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829213.879914:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973098496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829213.879928:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829213.879935:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829213/real 1787829213] req@ffff9c1a78861680 x1874668973098496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829213.879948:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973098496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829213.879953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829213.879955:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829213.879957:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973098496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829213.879961:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973098624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829213.879970:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829213.879975:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829213/real 1787829213] req@ffff9c1a78862580 x1874668973098624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829213.879983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973098624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829213.879988:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829213.879989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829213.879991:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973098624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829213.879995:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973098752:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829213.880006:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829213.880011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829213/real 1787829213] req@ffff9c1a78861e00 x1874668973098752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829213.880018:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973098752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829213.880027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829213.880029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829213.880031:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973098752:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829213.880034:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973098880:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829213.880043:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829213.880048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829213/real 1787829213] req@ffff9c1a78860780 x1874668973098880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829213.880054:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973098880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829213.880059:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829213.880061:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829213.880062:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973098880:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829214.391451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829214.391464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829214.647456:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829215.415420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829215.415443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829215.671494:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829216.439403:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829216.439425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829216.695468:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829217.463443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829217.463460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829217.719446:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829218.487440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829218.487446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829218.743541:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829218.999446:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829218.999454:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829218.999458:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829218.999466:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5669 00000100:00080000:2.0:1787829218.999475:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829218.999481:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829218.999521:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973099008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829218.999546:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829218.999553:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829218.999554:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829218.999557:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5669 00000100:00080000:2.0:1787829218.999560:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829218.999563:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829218.999571:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829218.999579:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973099136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829218.999586:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829218.999588:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829218.999589:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829218.999591:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5669 00000100:00080000:2.0:1787829218.999593:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829218.999595:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829218.999598:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829218.999604:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973099264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829218.999611:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829218.999614:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829218.999615:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829218.999618:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5669 00000100:00080000:2.0:1787829218.999620:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829218.999622:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829218.999626:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829218.999629:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973099008:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829218.999635:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973099392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829218.999644:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829218.999646:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829218.999647:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829218.999649:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5669 00000100:00080000:2.0:1787829218.999652:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829218.999653:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829218.999657:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829218.999665:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973099520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829218.999670:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829218.999673:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829218.999674:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829218.999682:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5669 00000100:00080000:2.0:1787829218.999685:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829218.999688:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829218.999692:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829218.999697:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829218.999706:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973099648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829218.999712:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829218.999715:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829218.999716:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829218.999719:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5669 00000100:00080000:2.0:1787829218.999723:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829218.999728:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829218/real 1787829218] req@ffff9c1a49d8f0c0 x1874668973099008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829218.999732:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829218.999742:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829218.999748:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973099008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829218.999758:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829218.999762:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973099776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829218.999763:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829218.999769:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973099008:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829218.999781:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973099136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829218.999800:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829218.999808:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829218/real 1787829218] req@ffff9c1a49d8e1c0 x1874668973099136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829218.999821:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973099136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829218.999827:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829218.999829:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829218.999832:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973099136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829218.999835:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973099264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829218.999845:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829218.999851:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829218/real 1787829218] req@ffff9c1a49d8c3c0 x1874668973099264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829218.999858:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973099264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829218.999863:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829218.999865:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829218.999867:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973099264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829218.999886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973099392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829218.999896:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829218.999902:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829218/real 1787829218] req@ffff9c1a49d8e580 x1874668973099392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829218.999910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973099392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829218.999925:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829218.999929:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829218.999932:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973099392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829218.999938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973099520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829218.999958:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829218.999969:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829218/real 1787829218] req@ffff9c1a49d8fc00 x1874668973099520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829218.999982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973099520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829218.999988:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829218.999990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829218.999993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973099520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829218.999997:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973099648:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829219.000012:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829219.000018:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829218/real 1787829219] req@ffff9c1a49d8ed00 x1874668973099648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829219.000026:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973099648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829219.000031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829219.000033:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829219.000035:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973099648:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829219.000039:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973099776:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829219.000049:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829219.000054:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829218/real 1787829219] req@ffff9c1a49d8d680 x1874668973099776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829219.000061:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973099776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829219.000066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829219.000068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829219.000070:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973099776:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829219.511405:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829219.511434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829219.767458:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829220.535416:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829220.535420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829220.791499:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829221.559403:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829221.559420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829221.815486:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829222.583441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829222.583441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829222.839512:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829223.607461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829223.607486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829223.863538:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829224.119429:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829224.119437:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829224.119441:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829224.119450:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5674 00000100:00080000:0.0:1787829224.119460:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829224.119465:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829224.119518:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973099904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829224.119547:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829224.119554:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829224.119555:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829224.119567:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5674 00000100:00080000:0.0:1787829224.119573:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829224.119579:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829224.119588:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973099904:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829224.119590:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829224.119610:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973100032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829224.119621:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829224.119624:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829224.119626:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829224.119629:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5674 00000100:00080000:0.0:1787829224.119633:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829224.119636:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829224.119640:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829224.119658:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829224.119664:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973100160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829224.119678:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829224.119683:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829224.119685:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829224.119691:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5674 00000100:00080000:0.0:1787829224.119696:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829224.119697:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff9c1a788612c0 x1874668973099904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829224.119705:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829224.119715:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829224.119721:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973099904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829224.119728:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973100288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829224.119752:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829224.119760:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829224.119762:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829224.119766:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829224.119770:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829224.119776:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5674 00000100:00080000:0.0:1787829224.119779:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829224.119781:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973099904:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829224.119782:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829224.119791:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829224.119821:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973100416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829224.119830:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973100032:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829224.119838:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829224.119844:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829224.119846:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829224.119853:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5674 00000100:00080000:0.0:1787829224.119859:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829224.119861:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829224.119885:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829224.119895:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829224.119896:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff9c1a78862940 x1874668973100032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829224.119916:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973100544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829224.119918:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973100032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829224.119967:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829224.119968:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829224.119974:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829224.119975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829224.119977:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829224.119981:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973100032:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829224.119984:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5674 00000100:00080000:0.0:1787829224.119990:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829224.119995:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829224.119995:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973100160:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787829224.120002:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829224.120032:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973100672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829224.120053:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829224.120074:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff9c1a78861a40 x1874668973100160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829224.120102:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973100160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829224.120108:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829224.120110:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829224.120113:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973100160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829224.120117:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973100288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829224.120132:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829224.120140:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff9c1a78860f00 x1874668973100288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829224.120169:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973100288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829224.120175:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829224.120177:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829224.120180:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973100288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829224.120190:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973100416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829224.120202:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829224.120208:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff9c1a788621c0 x1874668973100416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829224.120216:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973100416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829224.120221:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829224.120223:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829224.120225:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973100416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829224.120228:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973100544:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829224.120239:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829224.120244:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff9c1a78863840 x1874668973100544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829224.120269:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973100544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829224.120279:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829224.120281:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829224.120284:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973100544:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829224.120287:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973100672:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829224.120317:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829224.120324:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829224/real 1787829224] req@ffff9c1a78862d00 x1874668973100672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829224.120331:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973100672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829224.120336:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829224.120338:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829224.120340:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973100672:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829224.631439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829224.631460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829224.887473:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829225.655491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829225.655589:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829225.911530:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829226.679456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829226.679484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829226.935528:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829227.703420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829227.703499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829227.959491:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829228.727448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829228.727452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829228.983542:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829229.239441:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829229.239450:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829229.239453:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829229.239461:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5679 00000100:00080000:0.0:1787829229.239470:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829229.239477:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829229.239509:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973100800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829229.239527:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829229.239530:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829229.239531:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829229.239534:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5679 00000100:00080000:0.0:1787829229.239536:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829229.239538:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829229.239546:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829229.239557:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973100928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829229.239564:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829229.239567:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829229.239568:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829229.239571:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5679 00000100:00080000:0.0:1787829229.239573:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829229.239575:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829229.239580:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829229.239582:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973100800:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829229.239589:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973101056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829229.239598:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829229.239600:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829229.239601:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829229.239603:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5679 00000100:00080000:0.0:1787829229.239605:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829229.239607:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829229.239610:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829229.239619:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973101184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829229.239624:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829229.239626:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829229.239628:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829229.239630:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5679 00000100:00080000:0.0:1787829229.239632:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829229.239634:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829229.239638:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829229.239645:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973101312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829229.239652:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829229.239654:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829229.239655:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829229.239659:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829229.239663:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5679 00000100:00080000:0.0:1787829229.239666:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829229.239669:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829229.239672:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829229.239680:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973101440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829229.239686:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829229.239688:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829229.239689:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829229.239692:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5679 00000100:00080000:0.0:1787829229.239694:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829229.239696:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829229.239699:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829229.239704:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff9c1a78860b40 x1874668973100800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829229.239709:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973101568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829229.239745:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973100800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829229.239757:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829229.239763:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829229.239767:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973100800:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829229.239776:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973100928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829229.239801:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829229.239816:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff9c1a78862d00 x1874668973100928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829229.239829:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973100928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829229.239838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829229.239841:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829229.239845:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973100928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829229.239870:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973101056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829229.239892:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829229.239904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff9c1a78863840 x1874668973101056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829229.239916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973101056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829229.239924:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829229.239928:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829229.239931:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973101056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829229.239937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973101184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829229.239967:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829229.239978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff9c1a788621c0 x1874668973101184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829229.239991:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973101184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829229.240000:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829229.240003:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829229.240006:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973101184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829229.240012:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973101312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829229.240032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829229.240044:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff9c1a78860f00 x1874668973101312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829229.240057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973101312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829229.240064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829229.240068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829229.240071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973101312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829229.240077:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973101440:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829229.240097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829229.240109:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff9c1a78861a40 x1874668973101440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829229.240121:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973101440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829229.240130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829229.240133:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829229.240136:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973101440:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829229.240143:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973101568:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829229.240163:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829229.240175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829229/real 1787829229] req@ffff9c1a78862940 x1874668973101568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829229.240186:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973101568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829229.240194:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829229.240197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829229.240200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973101568:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829229.751418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829229.751425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829230.007493:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829230.775460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829230.775567:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829231.031535:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829231.799421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829231.799421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829232.055440:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829232.823495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829232.823497:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829233.079539:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829233.847468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829233.847500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829234.103500:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829234.359446:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829234.359454:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829234.359459:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.359467:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5684 00000100:00080000:0.0:1787829234.359477:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829234.359484:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829234.359515:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973101696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829234.359537:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829234.359540:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829234.359541:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.359544:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5684 00000100:00080000:0.0:1787829234.359547:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829234.359549:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829234.359557:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829234.359566:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973101824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829234.359578:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829234.359582:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787829234.359585:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973101696:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829234.359586:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.359596:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5684 00000100:00080000:0.0:1787829234.359601:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829234.359605:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829234.359611:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829234.359625:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973101952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829234.359636:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829234.359639:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829234.359641:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.359643:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5684 00000100:00080000:0.0:1787829234.359647:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829234.359649:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829234.359653:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829234.359666:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973102080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829234.359675:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829234.359676:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973101824:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829234.359680:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829234.359684:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.359687:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5684 00000100:00080000:0.0:1787829234.359690:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829234.359694:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829234.359697:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829234.359706:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973102208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829234.359712:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829234.359715:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973101952:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829234.359717:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829234.359721:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.359725:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5684 00000100:00080000:0.0:1787829234.359728:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829234.359731:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829234.359736:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829234.359742:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973102080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829234.359747:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973102336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829234.359756:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829234.359759:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829234.359760:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829234.359763:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5684 00000100:00080000:0.0:1787829234.359766:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829234.359767:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973102208:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829234.359771:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829234.359776:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829234.359786:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973102464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829234.359795:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973102336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829234.359814:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973102464:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829234.871492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829234.871559:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829235.127509:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829235.895427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829235.895427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829236.151477:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829236.919435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829236.919436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829237.175493:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829237.431610:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787829237.431701:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829237.431716:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829237.431749:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829237.431762:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829237.431775:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787829237.431780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829237] req@ffff9c1a78860780 x1874668973101824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829237.431785:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829237.431796:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829237.431804:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787829237.431804:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973101824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.431816:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829237.431821:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829237.431827:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973101824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829237.431835:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829237] req@ffff9c1a78861e00 x1874668973101952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829237.431848:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973101952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.431856:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829237.431860:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829237.431864:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973101952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829237.431871:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829237] req@ffff9c1a788612c0 x1874668973101696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829237.431884:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973101696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.431891:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829237.431894:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829237.431899:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973101696:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829237.431937:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829237] req@ffff9c1a78863c00 x1874668973102464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829237.431951:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973102464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.431959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829237.431963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829237.431966:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973102464:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829237.431972:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829237] req@ffff9c1a788630c0 x1874668973102336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829237.431983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973102336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.431989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829237.431992:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829237.431995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973102336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829237.432001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829237] req@ffff9c1a78862580 x1874668973102080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829237.432011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973102080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.432018:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829237.432023:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829237.432026:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973102080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829237.432032:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829234/real 1787829237] req@ffff9c1a78861680 x1874668973102208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829237.432042:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973102208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.432049:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829237.432052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829237.432055:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973102208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829237.943487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829237.943487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829238.199553:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829238.967441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829238.967452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829239.223494:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829239.479438:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829239.479449:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829239.479454:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829239.479467:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5689 00000100:00080000:0.0:1787829239.479478:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829239.479487:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829239.479537:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973102592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829239.479582:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829239.479591:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829239.479593:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829239.479599:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5689 00000100:00080000:0.0:1787829239.479605:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829239.479611:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829239.479625:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829239.479644:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973102592:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829239.479647:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973102720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829239.479673:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829239.479677:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829239.479679:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829239.479686:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5689 00000100:00080000:0.0:1787829239.479697:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829239.479702:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829239.479708:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829239.479724:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973102848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829239.479737:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829239.479742:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829239.479745:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829239.479751:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829239.479758:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5689 00000100:00080000:0.0:1787829239.479767:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829239.479775:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829239.479782:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829239.479792:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829239/real 1787829239] req@ffff9c1a78860000 x1874668973102592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829239.479803:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973102976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829239.479831:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829239.479835:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829239.479841:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829239.479844:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973102592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829239.479853:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5689 00000100:00080000:0.0:1787829239.479866:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829239.479877:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787829239.479878:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829239.479893:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829239.479894:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829239.479907:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973102592:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829239.479921:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973103104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829239.479939:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829239.479945:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973102720:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829239.479948:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829239.479953:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829239.479963:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5689 00000100:00080000:0.0:1787829239.479969:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829239.479976:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829239.479982:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829239.479991:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829239.480000:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973103232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829239.480013:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829239/real 1787829239] req@ffff9c1a78861680 x1874668973102720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829239.480041:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829239.480056:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829239.480059:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829239.480075:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5689 00000100:00080000:0.0:1787829239.480086:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829239.480092:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829239.480099:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973102720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829239.480104:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829239.480128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829239.480132:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973103360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829239.480140:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829239.480171:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973102720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829239.480184:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973102848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829239.480333:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829239.480387:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829239/real 1787829239] req@ffff9c1a78862580 x1874668973102848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829239.480401:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973102848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829239.480409:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829239.480413:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829239.480416:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973102848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829239.480423:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973102976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829239.480446:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829239.480457:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829239/real 1787829239] req@ffff9c1a788630c0 x1874668973102976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829239.480469:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973102976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829239.480483:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829239.480487:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829239.480489:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973102976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829239.480496:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973103104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829239.480516:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829239.480528:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829239/real 1787829239] req@ffff9c1a78863c00 x1874668973103104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829239.480541:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973103104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829239.480549:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829239.480553:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829239.480556:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973103104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829239.480579:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973103232:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829239.480602:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829239.480614:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829239/real 1787829239] req@ffff9c1a788612c0 x1874668973103232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829239.480627:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973103232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829239.480642:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829239.480646:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829239.480650:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973103232:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829239.480657:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973103360:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829239.480679:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829239.480691:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829239/real 1787829239] req@ffff9c1a78861e00 x1874668973103360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829239.480703:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973103360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829239.480712:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829239.480715:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829239.480718:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973103360:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829239.991509:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829239.991509:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829240.247568:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829241.015454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829241.015489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829241.271476:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829242.039490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829242.039490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829242.295678:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829243.063455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829243.063455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829243.319513:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829244.087472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829244.087478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829244.343582:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829244.599512:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829244.599523:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829244.599528:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829244.599539:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5694 00000100:00080000:3.0:1787829244.599551:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829244.599560:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829244.599599:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973103488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829244.599632:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829244.599636:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829244.599637:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829244.599641:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5694 00000100:00080000:3.0:1787829244.599645:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829244.599649:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829244.599658:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829244.599673:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973103616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829244.599678:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973103488:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829244.599688:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829244.599692:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829244.599694:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829244.599698:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5694 00000100:00080000:3.0:1787829244.599701:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829244.599704:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829244.599708:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829244.599718:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973103744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829244.599726:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829244.599729:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829244.599730:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829244.599732:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5694 00000100:00080000:3.0:1787829244.599741:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829244.599744:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829244.599747:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829244.599750:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829244.599761:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973103872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829244.599785:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829244/real 1787829244] req@ffff9c1a7895e580 x1874668973103488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829244.599800:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973103488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829244.599807:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829244.599810:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829244.599813:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973103488:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829244.599832:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973103616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829244.599849:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829244.599854:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829244.599860:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829244/real 1787829244] req@ffff9c1a7895c000 x1874668973103616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829244.599862:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829244.599867:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829244.599878:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973103616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829244.599885:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829244.599887:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829244.599889:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973103616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829244.599893:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973103744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829244.599905:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829244.599912:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829244/real 1787829244] req@ffff9c1a7895d2c0 x1874668973103744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829244.599919:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973103744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829244.599924:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829244.599926:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829244.599928:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973103744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829244.599932:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973103872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829244.599944:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829244.599951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829244/real 1787829244] req@ffff9c1a7895f480 x1874668973103872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829244.599960:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973103872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829244.599965:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829244.599967:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829244.599969:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973103872:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829244.599983:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5694 00000100:00080000:3.0:1787829244.599990:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829244.599994:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829244.600000:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829244.600050:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973104000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829244.600065:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829244.600068:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829244.600070:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829244.600073:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5694 00000100:00080000:3.0:1787829244.600077:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829244.600080:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829244.600085:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973104000:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829244.600085:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829244.600120:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973104128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829244.600131:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829244.600137:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829244.600158:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829244.600161:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829244.600167:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829244/real 1787829244] req@ffff9c1a7895cf00 x1874668973104000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829244.600171:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5694 00000100:00080000:3.0:1787829244.600182:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829244.600191:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973104000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829244.600191:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829244.600201:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829244.600205:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829244.600211:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829244.600215:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973104000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829244.600220:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973104256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829244.600233:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973104128:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829244.600257:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829244.600267:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829244/real 1787829244] req@ffff9c1a7895c3c0 x1874668973104128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829244.600276:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973104128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829244.600288:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829244.600291:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829244.600293:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973104128:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829244.600301:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973104256:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829244.600313:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829244.600320:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829244/real 1787829244] req@ffff9c1a7895e940 x1874668973104256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829244.600329:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973104256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829244.600335:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829244.600337:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829244.600340:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973104256:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829245.111474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829245.111490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829245.367493:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829246.135468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829246.135468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829246.391477:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829247.159445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829247.159463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829247.415556:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829248.183445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829248.183471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829248.439538:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829249.207441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829249.207447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829249.463632:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829249.719476:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829249.719487:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829249.719491:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829249.719511:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5700 00000100:00080000:3.0:1787829249.719522:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829249.719530:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829249.719566:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973104384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829249.719602:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829249.719607:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829249.719609:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829249.719613:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5700 00000100:00080000:3.0:1787829249.719618:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829249.719623:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829249.719633:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829249.719643:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973104384:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829249.719650:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973104512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829249.719665:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829249.719669:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829249.719671:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829249.719676:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5700 00000100:00080000:3.0:1787829249.719681:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829249.719686:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829249.719691:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829249.719707:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973104640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829249.719708:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829249.719722:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829249.719727:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829249.719729:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829249.719734:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5700 00000100:00080000:3.0:1787829249.719739:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829249.719743:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829249/real 1787829249] req@ffff9c1a7895c780 x1874668973104384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829249.719745:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829249.719756:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829249.719764:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973104384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829249.719773:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973104768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829249.719775:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829249.719782:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829249.719786:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973104384:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829249.719799:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829249.719804:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829249.719806:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973104512:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829249.719808:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829249.719816:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5700 00000100:00080000:3.0:1787829249.719822:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829249.719828:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829249.719829:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829249.719836:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829249.719841:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829249/real 1787829249] req@ffff9c1a7895cb40 x1874668973104512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829249.719850:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973104896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829249.719855:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973104512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829249.719870:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829249.719871:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829249.719874:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829249.719879:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973104512:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829249.719881:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829249.719885:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829249.719886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973104640:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829249.719896:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5700 00000100:00080000:3.0:1787829249.719903:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829249.719905:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829249.719920:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829249.719921:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829249/real 1787829249] req@ffff9c1a7895da40 x1874668973104640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829249.719927:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829249.719942:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973105024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829249.719946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973104640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829249.719959:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829249.719968:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829249.719970:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829249.719970:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829249.719979:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5700 00000100:00080000:0.0:1787829249.719982:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829249.719984:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829249.719988:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973104640:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829249.719991:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829249.719997:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829249.720000:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973104768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829249.720008:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973105152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829249.720024:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829249.720035:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829249/real 1787829249] req@ffff9c1a7895e1c0 x1874668973104768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829249.720044:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973104768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829249.720050:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829249.720052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829249.720054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973104768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829249.720065:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973104896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829249.720097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829249.720105:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829249/real 1787829249] req@ffff9c1a7895fc00 x1874668973104896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829249.720113:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973104896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829249.720119:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829249.720121:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829249.720123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973104896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829249.720126:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973105024:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829249.720138:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829249.720144:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829249/real 1787829249] req@ffff9c1a7895f840 x1874668973105024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829249.720151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973105024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829249.720157:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829249.720158:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829249.720160:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973105024:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829249.720163:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973105152:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829249.720181:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829249.720187:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829249/real 1787829249] req@ffff9c1a7895f0c0 x1874668973105152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829249.720195:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973105152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829249.720200:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829249.720202:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829249.720204:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973105152:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829250.231442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829250.231443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829250.487527:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829251.255454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829251.255455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829251.511538:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829252.279448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829252.279481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829252.535580:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829253.303537:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829253.303537:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829253.559447:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829254.327428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829254.327483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829254.583517:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829254.839461:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829254.839472:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829254.839477:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829254.839489:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5705 00000100:00080000:3.0:1787829254.839501:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829254.839508:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829254.839552:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973105280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829254.839578:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829254.839581:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829254.839583:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829254.839586:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5705 00000100:00080000:3.0:1787829254.839589:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829254.839592:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829254.839602:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829254.839615:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973105408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829254.839623:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829254.839629:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973105280:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829254.839629:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829254.839631:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829254.839639:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5705 00000100:00080000:3.0:1787829254.839642:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829254.839644:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829254.839647:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829254.839655:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973105536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829254.839668:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829254.839671:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829254.839673:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829254.839675:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5705 00000100:00080000:3.0:1787829254.839677:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829254.839680:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829254.839683:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829254.839689:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973105664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829254.839695:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829254.839698:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829254.839700:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829254.839702:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5705 00000100:00080000:3.0:1787829254.839704:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829254.839706:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829254.839709:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829254.839714:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829254.839720:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973105792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829254.839728:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829254.839730:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829254.839732:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829254.839735:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5705 00000100:00080000:3.0:1787829254.839739:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829254.839742:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829254.839745:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829254.839753:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973105920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829254.839760:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829254.839763:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829254/real 1787829254] req@ffff9c1a7895de00 x1874668973105280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829254.839764:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829254.839768:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829254.839776:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5705 00000100:00080000:3.0:1787829254.839781:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829254.839787:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829254.839792:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829254.839796:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973105280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829254.839809:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973106048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829254.839813:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829254.839824:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829254.839830:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973105280:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829254.839859:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973105408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829254.839881:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829254.839891:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829254/real 1787829254] req@ffff9c1a7895f0c0 x1874668973105408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829254.839901:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973105408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829254.839907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829254.839915:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829254.839918:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973105408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829254.839922:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973105536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829254.839936:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829254.839943:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829254/real 1787829254] req@ffff9c1a7895f840 x1874668973105536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829254.839951:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973105536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829254.839961:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829254.839963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829254.839965:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973105536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829254.839969:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973105664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829254.839980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829254.839985:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829254/real 1787829254] req@ffff9c1a7895fc00 x1874668973105664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829254.839993:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973105664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829254.839999:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829254.840000:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829254.840002:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973105664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829254.840006:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973105792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829254.840017:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829254.840023:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829254/real 1787829254] req@ffff9c1a7895e1c0 x1874668973105792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829254.840030:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973105792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829254.840051:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829254.840054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829254.840057:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973105792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829254.840062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973105920:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829254.840079:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829254.840087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829254/real 1787829254] req@ffff9c1a7895da40 x1874668973105920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829254.840095:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973105920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829254.840101:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829254.840103:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829254.840105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973105920:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829254.840108:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973106048:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829254.840120:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829254.840130:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829254/real 1787829254] req@ffff9c1a7895cb40 x1874668973106048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829254.840143:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973106048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829254.840152:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829254.840155:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829254.840159:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973106048:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829255.351476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829255.351476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829255.607552:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829256.375421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829256.375429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829256.631466:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829257.399442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829257.399442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829257.655530:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829258.423432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829258.423432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829258.679550:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829259.447464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829259.447479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829259.703556:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829259.959530:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829259.959545:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829259.959551:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829259.959564:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5710 00000100:00080000:2.0:1787829259.959578:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829259.959587:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829259.959636:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973106176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829259.959676:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829259.959681:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829259.959683:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829259.959689:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5710 00000100:00080000:2.0:1787829259.959694:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829259.959700:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829259.959712:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829259.959731:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973106304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829259.959742:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973106176:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829259.959754:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829259.959760:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829259.959762:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829259.959767:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5710 00000100:00080000:2.0:1787829259.959771:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829259.959776:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829259.959782:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829259.959798:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973106432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829259.959812:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829259.959817:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829259.959820:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829259.959825:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5710 00000100:00080000:2.0:1787829259.959829:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829259.959833:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787829259.959841:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787829259.959843:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829259.959864:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973106560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829259.959878:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829259.959882:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829259.959885:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829259.959891:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5710 00000100:00080000:2.0:1787829259.959894:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829259.959898:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829259.959903:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829259.959914:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829259/real 1787829259] req@ffff9c1a49d8d680 x1874668973106176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829259.959921:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973106688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829259.959964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829259.959968:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829259.959972:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829259.959975:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973106176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829259.959983:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5710 00000100:00080000:2.0:1787829259.959993:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829259.960031:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787829259.960039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829259.960041:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829259.960054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829259.960063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973106176:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829259.960067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973106816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829259.960084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829259.960089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829259.960092:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829259.960098:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5710 00000100:00080000:2.0:1787829259.960105:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829259.960109:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973106304:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829259.960115:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829259.960127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829259.960147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973106944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829259.960174:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829259.960220:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829259/real 1787829259] req@ffff9c1a49d8ed00 x1874668973106304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829259.960248:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973106304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829259.960268:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829259.960275:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829259.960283:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973106304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829259.960297:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973106432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829259.960337:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829259.960389:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829259/real 1787829259] req@ffff9c1a49d8fc00 x1874668973106432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829259.960405:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973106432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829259.960416:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829259.960420:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829259.960425:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973106432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829259.960432:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973106560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829259.960458:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829259.960470:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829259/real 1787829259] req@ffff9c1a49d8e580 x1874668973106560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829259.960484:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973106560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829259.960495:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829259.960498:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829259.960502:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973106560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829259.960509:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973106688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829259.960534:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829259.960554:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829259/real 1787829259] req@ffff9c1a49d8c3c0 x1874668973106688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829259.960593:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973106688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829259.960611:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829259.960617:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829259.960625:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973106688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829259.960637:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973106816:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829259.960668:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829259.960684:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829259/real 1787829259] req@ffff9c1a49d8e1c0 x1874668973106816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829259.960699:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973106816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829259.960710:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829259.960714:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829259.960718:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973106816:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829259.960744:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973106944:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829259.960765:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829259.960777:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829259/real 1787829259] req@ffff9c1a49d8f0c0 x1874668973106944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829259.960791:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973106944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829259.960810:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829259.960813:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829259.960817:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973106944:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829260.471442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829260.471446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829260.727494:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829261.495469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829261.495506:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829261.751617:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829262.519418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829262.519418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829262.775555:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829263.543459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829263.543562:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829263.799600:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829264.567431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829264.567436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829264.823498:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829265.079416:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829265.079424:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829265.079428:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829265.079436:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5715 00000100:00080000:2.0:1787829265.079446:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829265.079451:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829265.079484:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973107072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829265.079514:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829265.079517:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829265.079518:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829265.079521:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5715 00000100:00080000:2.0:1787829265.079523:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829265.079526:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829265.079533:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829265.079543:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973107200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829265.079552:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829265.079554:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829265.079556:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829265.079561:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973107072:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829265.079562:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5715 00000100:00080000:2.0:1787829265.079567:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829265.079575:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829265.079578:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829265.079587:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973107328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829265.079595:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829265.079598:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829265.079600:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829265.079602:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5715 00000100:00080000:2.0:1787829265.079605:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829265.079607:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829265.079610:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829265.079621:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973107456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829265.079624:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829265.079636:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829265.079640:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829265.079643:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829265.079647:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5715 00000100:00080000:2.0:1787829265.079653:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829265.079657:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829265.079663:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829265.079671:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff9c1a49d8cf00 x1874668973107072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829265.079677:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973107584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829265.079694:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829265.079696:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829265.079699:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829265.079700:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973107072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829265.079704:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5715 00000100:00080000:2.0:1787829265.079710:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829265.079716:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787829265.079717:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829265.079722:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829265.079723:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829265.079729:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973107072:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829265.079733:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973107712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829265.079748:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829265.079750:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829265.079752:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829265.079754:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5715 00000100:00080000:2.0:1787829265.079756:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829265.079760:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829265.079764:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973107200:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787829265.079767:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829265.079785:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973107840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829265.079790:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829265.079812:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff9c1a49d8cb40 x1874668973107200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829265.079824:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973107200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829265.079832:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829265.079835:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829265.079837:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973107200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829265.079842:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973107328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829265.079853:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829265.079860:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff9c1a49d8da40 x1874668973107328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829265.079867:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973107328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829265.079872:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829265.079874:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829265.079876:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973107328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829265.079879:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973107456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829265.079889:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829265.079895:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff9c1a49d8de00 x1874668973107456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829265.079902:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973107456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829265.079907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829265.079909:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829265.079911:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973107456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829265.079914:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973107584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829265.079923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829265.079929:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff9c1a49d8c780 x1874668973107584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829265.079935:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973107584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829265.079940:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829265.079942:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829265.079944:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973107584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829265.079947:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973107712:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829265.079957:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829265.079962:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff9c1a49d8c000 x1874668973107712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829265.079987:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973107712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829265.079992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829265.079994:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829265.079996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973107712:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829265.080011:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973107840:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829265.080025:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829265.080033:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829265/real 1787829265] req@ffff9c1a49d8cf00 x1874668973107840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829265.080040:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973107840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829265.080046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829265.080048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829265.080050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973107840:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829265.591472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829265.591472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829265.847590:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829266.615529:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829266.615529:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829266.871535:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829267.639468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829267.639472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829267.895510:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829268.663451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829268.663461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829268.919558:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829269.687439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829269.687449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829269.943614:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829270.199440:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829270.199447:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829270.199451:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829270.199459:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5720 00000100:00080000:3.0:1787829270.199468:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829270.199473:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829270.199502:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973107968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829270.199524:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829270.199527:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829270.199528:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829270.199531:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5720 00000100:00080000:3.0:1787829270.199533:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829270.199541:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829270.199548:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829270.199559:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973108096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829270.199568:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973107968:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829270.199575:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829270.199580:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829270.199582:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829270.199587:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5720 00000100:00080000:3.0:1787829270.199592:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829270.199597:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829270.199603:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829270.199616:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973108224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829270.199625:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829270.199626:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829270.199630:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829270.199632:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829270.199636:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5720 00000100:00080000:3.0:1787829270.199641:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829270.199645:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829270.199648:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829270.199656:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973108352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829270.199663:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829270.199665:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829270.199669:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff9c1a7895c780 x1874668973107968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829270.199669:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829270.199677:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5720 00000100:00080000:3.0:1787829270.199682:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829270.199689:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829270.199693:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829270.199697:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973107968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829270.199703:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973108480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829270.199712:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829270.199714:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829270.199718:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829270.199739:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973107968:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829270.199744:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829270.199748:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829270.199753:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5720 00000100:00080000:3.0:1787829270.199756:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829270.199760:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829270.199763:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973108096:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829270.199767:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829270.199782:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973108608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829270.199793:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829270.199805:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829270.199808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829270.199810:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff9c1a7895e940 x1874668973108096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829270.199811:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829270.199820:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5720 00000100:00080000:3.0:1787829270.199827:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829270.199833:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829270.199837:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829270.199838:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973108096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829270.199848:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829270.199850:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973108736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829270.199854:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829270.199862:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973108096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829270.199875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973108224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829270.199893:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829270.199902:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff9c1a7895c3c0 x1874668973108224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829270.199911:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973108224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829270.199916:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829270.199918:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829270.199920:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973108224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829270.199925:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973108352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829270.199956:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829270.199976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff9c1a7895cf00 x1874668973108352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829270.199985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973108352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829270.199992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829270.199994:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829270.199997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973108352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829270.200001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973108480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829270.200013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829270.200019:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff9c1a7895f480 x1874668973108480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829270.200027:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973108480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829270.200033:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829270.200035:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829270.200037:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973108480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829270.200050:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973108608:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829270.200063:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829270.200069:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff9c1a7895d2c0 x1874668973108608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829270.200077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973108608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829270.200083:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829270.200085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829270.200088:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973108608:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829270.200092:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973108736:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829270.200102:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829270.200108:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829270/real 1787829270] req@ffff9c1a7895c000 x1874668973108736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829270.200116:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973108736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829270.200150:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829270.200152:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829270.200153:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973108736:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829270.711422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829270.711422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829270.967550:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829271.735482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829271.735482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829271.991542:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829272.759483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829272.759483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829273.015519:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829273.783433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829273.783435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829274.039556:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829274.807498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829274.807498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829275.063547:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829275.319461:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829275.319472:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829275.319477:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829275.319488:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5725 00000100:00080000:0.0:1787829275.319501:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829275.319508:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829275.319570:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973108864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829275.319603:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829275.319607:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829275.319615:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829275.319619:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5725 00000100:00080000:0.0:1787829275.319623:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829275.319643:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829275.319654:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829275.319668:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973108992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829275.319670:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973108864:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829275.319684:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829275.319687:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829275.319688:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829275.319691:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5725 00000100:00080000:0.0:1787829275.319693:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829275.319696:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829275.319699:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829275.319707:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973109120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829275.319715:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829275.319718:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829275.319720:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829275.319722:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5725 00000100:00080000:0.0:1787829275.319725:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829275.319728:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829275.319732:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829275.319739:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973109248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829275.319746:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829275.319748:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829275.319750:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829275.319754:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829275.319758:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5725 00000100:00080000:0.0:1787829275.319762:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829275.319764:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829275.319767:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829275.319776:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973109376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829275.319782:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829275.319784:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829275.319786:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829275.319789:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5725 00000100:00080000:0.0:1787829275.319791:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829275.319794:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829275.319798:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829275.319802:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff9c1a78860780 x1874668973108864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829275.319809:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973109504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829275.319823:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829275.319826:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829275.319829:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973108864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829275.319830:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829275.319838:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5725 00000100:00080000:0.0:1787829275.319844:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829275.319847:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829275.319848:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829275.319855:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829275.319856:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829275.319863:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973108864:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829275.319867:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973109632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829275.319891:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973108992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829275.319926:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829275.319936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff9c1a78862940 x1874668973108992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829275.319946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973108992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829275.319952:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829275.319954:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829275.319974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973108992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829275.319980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973109120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829275.319996:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829275.320003:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff9c1a78861a40 x1874668973109120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829275.320013:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973109120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829275.320020:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829275.320022:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829275.320024:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973109120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829275.320029:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973109248:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829275.320068:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829275.320080:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff9c1a78860f00 x1874668973109248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829275.320095:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973109248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829275.320114:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829275.320117:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829275.320120:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973109248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829275.320125:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973109376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829275.320158:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829275.320166:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff9c1a788621c0 x1874668973109376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829275.320174:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973109376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829275.320180:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829275.320182:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829275.320184:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973109376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829275.320187:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973109504:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829275.320198:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829275.320204:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff9c1a78863840 x1874668973109504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829275.320211:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973109504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829275.320216:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829275.320218:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829275.320220:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973109504:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829275.320224:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973109632:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829275.320234:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829275.320239:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829275/real 1787829275] req@ffff9c1a78862d00 x1874668973109632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829275.320246:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973109632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829275.320252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829275.320254:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829275.320255:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973109632:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829275.831466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829275.831467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829276.087534:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829276.855461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829276.855461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829277.111541:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829277.879470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829277.879471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829278.135502:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829278.903455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829278.903473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829279.159540:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829279.927463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829279.927488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829280.183510:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829280.439528:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829280.439538:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829280.439544:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829280.439559:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5730 00000100:00080000:3.0:1787829280.439572:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829280.439581:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829280.439622:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973109760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829280.439655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829280.439660:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829280.439662:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829280.439668:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5730 00000100:00080000:3.0:1787829280.439675:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829280.439683:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829280.439695:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829280.439705:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973109760:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829280.439722:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973109888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829280.439746:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829280.439751:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829280.439756:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829280.439762:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5730 00000100:00080000:3.0:1787829280.439771:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829280.439777:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787829280.439784:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829280.439787:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829280.439833:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff9c1a7895e580 x1874668973109760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829280.439854:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973109760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829280.439883:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829280.439904:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829280.439910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973109760:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829280.439946:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973109888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829280.440002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829280.440025:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff9c1a7895c000 x1874668973109888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829280.440046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973109888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829280.440057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829280.440062:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829280.440066:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973109888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829280.440185:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973110016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829280.440217:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829280.440277:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829280.440281:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829280.440282:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973110016:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829280.440294:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5730 00000100:00080000:3.0:1787829280.440304:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829280.440313:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829280.440323:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829280.440336:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829280.440387:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff9c1a7895d2c0 x1874668973110016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829280.440404:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973110144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829280.440409:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973110016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829280.440439:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829280.440450:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829280.440455:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829280.440459:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973110016:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829280.440462:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829280.440467:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829280.440474:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5730 00000100:00080000:3.0:1787829280.440480:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829280.440481:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973110144:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829280.440490:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829280.440499:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829280.440514:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829280.440520:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973110272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829280.440533:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff9c1a7895f480 x1874668973110144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829280.440537:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829280.440549:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829280.440553:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829280.440562:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5730 00000100:00100000:0.0:1787829280.440564:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973110144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829280.440569:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829280.440580:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829280.440591:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829280.440594:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829280.440602:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829280.440607:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973110144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829280.440611:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973110400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829280.440624:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973110272:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829280.440630:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829280.440636:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829280.440639:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829280.440646:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5730 00000100:00080000:3.0:1787829280.440652:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829280.440654:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829280.440677:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829280.440687:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829280.440694:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff9c1a7895cf00 x1874668973110272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829280.440709:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973110528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829280.440718:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973110272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829280.440739:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829280.440743:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829280.440749:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973110272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829280.440768:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973110400:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829280.440795:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829280.440808:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff9c1a7895c3c0 x1874668973110400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829280.440822:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973110400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829280.440832:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829280.440836:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829280.440841:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973110400:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829280.440848:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973110528:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829280.440910:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829280.440927:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829280/real 1787829280] req@ffff9c1a7895e940 x1874668973110528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829280.440945:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973110528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829280.440957:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829280.440962:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829280.440967:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973110528:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829280.951507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829280.951508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829281.207593:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829281.975456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829281.975456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829282.231471:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829282.999472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829282.999476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829283.255594:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829284.023413:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829284.023430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829284.279569:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829285.047480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829285.047480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829285.303608:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829285.559537:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829285.559548:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829285.559553:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829285.559582:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5735 00000100:00080000:3.0:1787829285.559596:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829285.559603:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829285.559677:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973110656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829285.559727:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829285.559730:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829285.559732:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829285.559735:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5735 00000100:00080000:3.0:1787829285.559739:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829285.559742:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829285.559754:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829285.559783:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973110784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829285.559794:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829285.559798:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829285.559801:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829285.559804:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5735 00000100:00080000:3.0:1787829285.559807:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829285.559811:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829285.559814:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829285.559840:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973110912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829285.559853:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829285.559877:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829285.559879:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829285.559881:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973110656:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829285.559885:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5735 00000100:00080000:3.0:1787829285.559897:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829285.559906:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829285.559912:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829285.559924:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973111040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829285.559930:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829285.559933:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829285.559934:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829285.559937:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5735 00000100:00080000:3.0:1787829285.559939:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829285.559943:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829285.559948:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829285.559960:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973111168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829285.559967:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829285.559970:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829285.559972:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829285.559974:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5735 00000100:00080000:3.0:1787829285.559977:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829285.559979:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829285.559983:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829285.559995:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973111296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829285.560003:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829285.560004:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829285.560013:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829285.560017:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829285.560026:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5735 00000100:00080000:3.0:1787829285.560031:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829285.560036:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829285.560043:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829285.560063:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973111424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829285.560085:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829285/real 1787829285] req@ffff9c1a7895c780 x1874668973110656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829285.560111:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973110656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829285.560125:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829285.560131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829285.560137:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973110656:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829285.560148:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973110784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829285.560181:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829285.560208:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829285/real 1787829285] req@ffff9c1a7895cb40 x1874668973110784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829285.560224:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973110784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829285.560235:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829285.560239:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829285.560244:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973110784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829285.560273:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973110912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829285.560296:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829285.560314:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829285/real 1787829285] req@ffff9c1a7895da40 x1874668973110912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829285.560328:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973110912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829285.560408:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829285.560413:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829285.560417:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973110912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829285.560426:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973111040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829285.560461:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829285.560476:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829285/real 1787829285] req@ffff9c1a7895e1c0 x1874668973111040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829285.560491:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973111040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829285.560511:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829285.560515:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829285.560519:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973111040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829285.560526:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973111168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829285.560545:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829285.560556:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829285/real 1787829285] req@ffff9c1a7895fc00 x1874668973111168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829285.560588:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973111168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829285.560604:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829285.560610:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829285.560617:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973111168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829285.560630:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973111296:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829285.560683:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829285.560715:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829285/real 1787829285] req@ffff9c1a7895f840 x1874668973111296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829285.560741:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973111296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829285.560758:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829285.560775:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829285.560782:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973111296:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829285.560791:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973111424:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829285.560852:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829285.560882:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829285/real 1787829285] req@ffff9c1a7895f0c0 x1874668973111424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829285.560898:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973111424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829285.560909:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829285.560913:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829285.560916:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973111424:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829286.071405:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829286.071426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829286.327639:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829287.095459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829287.095460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829287.351623:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829288.119439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829288.119453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829288.375492:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829289.143431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829289.143431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829289.399467:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829290.167483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829290.167483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829290.423483:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829290.679454:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829290.679465:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829290.679471:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829290.679483:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5741 00000100:00080000:3.0:1787829290.679495:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829290.679503:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829290.679543:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973111552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829290.679587:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829290.679591:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829290.679594:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829290.679599:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5741 00000100:00080000:3.0:1787829290.679604:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829290.679610:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829290.679619:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829290.679622:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973111552:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829290.679635:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973111680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829290.679646:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829290.679648:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829290.679650:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829290.679653:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5741 00000100:00080000:3.0:1787829290.679656:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829290.679659:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829290.679663:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829290.679671:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973111808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829290.679700:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829290.679708:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829290.679711:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829290.679712:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829290.679716:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5741 00000100:00080000:3.0:1787829290.679721:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829290.679723:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829290.679727:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829290.679737:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973111936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829290.679741:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829290/real 1787829290] req@ffff9c1a7895de00 x1874668973111552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829290.679753:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829290.679758:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829290.679760:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829290.679763:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5741 00000100:00080000:3.0:1787829290.679765:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829290.679767:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829290.679770:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829290.679777:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973111552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829290.679778:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973112064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829290.679825:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829290.679825:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829290.679833:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829290.679834:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829290.679839:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829290.679840:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973111552:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829290.679853:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5741 00000100:00080000:3.0:1787829290.679860:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829290.679866:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829290.679870:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973111680:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829290.679875:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829290.679895:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973112192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829290.679897:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829290.679910:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829290/real 1787829290] req@ffff9c1a7895f0c0 x1874668973111680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829290.679912:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829290.679923:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829290.679928:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829290.679933:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973111680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829290.679938:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5741 00000100:00080000:0.0:1787829290.679944:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829290.679947:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829290.679950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829290.679955:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973111680:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829290.679958:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829290.679963:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973111808:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829290.679969:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829290.679980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829290.679990:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973112320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829290.679991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829290/real 1787829290] req@ffff9c1a7895f840 x1874668973111808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829290.680041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973111808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829290.680050:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829290.680052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829290.680055:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973111808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829290.680062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973111936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829290.680085:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829290.680094:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829290/real 1787829290] req@ffff9c1a7895fc00 x1874668973111936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829290.680103:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973111936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829290.680110:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829290.680112:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829290.680115:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973111936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829290.680119:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973112064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829290.680131:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829290.680137:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829290/real 1787829290] req@ffff9c1a7895e1c0 x1874668973112064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829290.680145:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973112064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829290.680151:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829290.680153:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829290.680156:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973112064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829290.680168:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973112192:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829290.680180:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829290.680186:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829290/real 1787829290] req@ffff9c1a7895da40 x1874668973112192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829290.680195:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973112192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829290.680201:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829290.680203:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829290.680205:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973112192:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829290.680209:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973112320:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829290.680218:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829290.680225:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829290/real 1787829290] req@ffff9c1a7895cb40 x1874668973112320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829290.680233:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973112320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829290.680238:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829290.680240:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829290.680242:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973112320:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829291.191478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829291.191478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829291.447540:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829292.215456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829292.215456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829292.471484:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829293.239435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829293.239435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829293.495512:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829294.263520:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829294.263535:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829294.519549:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829295.287445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829295.287450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829295.543545:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829295.799491:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829295.799503:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829295.799509:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829295.799521:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5746 00000100:00080000:0.0:1787829295.799533:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829295.799542:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829295.799592:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973112448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829295.799621:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829295.799624:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829295.799625:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829295.799629:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5746 00000100:00080000:0.0:1787829295.799633:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829295.799637:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829295.799648:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829295.799664:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973112448:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829295.799665:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973112576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829295.799681:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829295.799683:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829295.799684:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829295.799687:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5746 00000100:00080000:0.0:1787829295.799688:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829295.799691:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829295.799694:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829295.799703:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973112704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829295.799715:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829295.799719:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829295.799720:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829295.799723:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5746 00000100:00080000:0.0:1787829295.799726:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829295.799728:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829295.799731:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829295.799738:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973112832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829295.799744:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829295.799746:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829295.799748:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829295.799767:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5746 00000800:00000100:3.0:1787829295.799771:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829295.799772:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829295.799777:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829295.799781:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829295.799790:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973112960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829295.799798:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829295.799800:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829295.799802:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829295.799805:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5746 00000100:00080000:0.0:1787829295.799807:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829295.799810:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829295.799815:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829295.799826:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973113088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829295.799830:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829295/real 1787829295] req@ffff9c1a78860b40 x1874668973112448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829295.799838:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829295.799848:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829295.799850:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829295.799854:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5746 00000100:00080000:0.0:1787829295.799856:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829295.799862:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829295.799862:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973112448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829295.799871:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829295.799884:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829295.799887:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973113216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829295.799899:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829295.799910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973112448:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829295.799938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973112576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829295.799959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829295.799969:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829295/real 1787829295] req@ffff9c1a78862d00 x1874668973112576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829295.799978:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973112576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829295.799984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829295.799987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829295.799989:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973112576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829295.799993:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973112704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829295.800004:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829295.800010:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829295/real 1787829295] req@ffff9c1a78863840 x1874668973112704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829295.800017:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973112704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829295.800022:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829295.800024:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829295.800026:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973112704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829295.800030:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973112832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829295.800040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829295.800046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829295/real 1787829295] req@ffff9c1a788621c0 x1874668973112832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829295.800054:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973112832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829295.800060:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829295.800061:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829295.800063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973112832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829295.800067:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973112960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829295.800077:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829295.800082:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829295/real 1787829295] req@ffff9c1a78860f00 x1874668973112960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829295.800090:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973112960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829295.800095:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829295.800097:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829295.800098:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973112960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829295.800102:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973113088:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829295.800112:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829295.800117:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829295/real 1787829295] req@ffff9c1a78861a40 x1874668973113088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829295.800124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973113088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829295.800130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829295.800132:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829295.800135:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973113088:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829295.800138:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973113216:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829295.800147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829295.800152:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829295/real 1787829295] req@ffff9c1a78862940 x1874668973113216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829295.800158:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973113216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829295.800163:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829295.800165:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829295.800167:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973113216:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829296.311484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829296.311490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829296.567470:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829297.335479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829297.335485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829297.591539:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829298.359445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829298.359477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829298.615616:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829299.383502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829299.383509:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829299.639515:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829300.407453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829300.407500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829300.663581:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829300.919544:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829300.919558:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829300.919564:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829300.919578:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5751 00000100:00080000:3.0:1787829300.919594:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829300.919604:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829300.919660:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973113344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829300.919736:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829300.919741:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829300.919744:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829300.919749:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5751 00000100:00080000:3.0:1787829300.919754:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829300.919759:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829300.919770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973113344:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787829300.919779:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829300.919798:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973113472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829300.919811:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829300.919815:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829300.919817:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829300.919821:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5751 00000100:00080000:3.0:1787829300.919826:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829300.919831:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829300.919837:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829300.919848:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973113600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829300.919859:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829300.919863:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829300.919866:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829300.919870:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5751 00000100:00080000:3.0:1787829300.919874:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829300.919878:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829300.919884:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973113472:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829300.919886:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829300.919905:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973113728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829300.919920:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973113600:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829300.919921:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829300.919928:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829300.919930:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829300.919936:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5751 00000100:00080000:3.0:1787829300.919941:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829300.919946:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829300.919955:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829300.919965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973113728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829300.919974:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973113856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829300.919990:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829300.919995:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829300.919997:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973113856:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829300.919998:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829300.920007:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5751 00000100:00080000:3.0:1787829300.920012:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829300.920021:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829300.920028:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829300.920043:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973113984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829300.920065:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829300.920068:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829300.920070:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829300.920074:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5751 00000100:00080000:3.0:1787829300.920078:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829300.920082:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829300.920087:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829300.920112:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973114112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829300.920119:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973113984:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829300.920157:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973114112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829301.431432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829301.431449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829301.687557:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829302.455491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829302.455491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829302.711561:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829303.479429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829303.479459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829303.735492:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829303.991584:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787829303.991622:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829303.991635:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829303.991672:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829303.991682:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829303.991689:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829303.991715:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829303.991725:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829303.991736:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787829303.991757:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829300/real 1787829303] req@ffff9c1a7895d2c0 x1874668973113984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829303.991786:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973113984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829303.991799:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829303.991804:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829303.991809:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973113984:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829303.991817:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829300/real 1787829303] req@ffff9c1a7895f480 x1874668973113856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829303.991829:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973113856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829303.991838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829303.991842:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829303.991846:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973113856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829303.991853:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829300/real 1787829303] req@ffff9c1a7895cf00 x1874668973113728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829303.991867:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973113728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829303.991894:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829303.991898:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829303.991901:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973113728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829303.991908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829300/real 1787829303] req@ffff9c1a7895e940 x1874668973113472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829303.991920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973113472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829303.991929:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829303.991932:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829303.991935:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973113472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829303.991941:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829300/real 1787829303] req@ffff9c1a7895c3c0 x1874668973113600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829303.991953:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973113600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829303.991961:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829303.991964:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829303.991967:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973113600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829303.991974:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829300/real 1787829303] req@ffff9c1a7895c780 x1874668973113344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829303.991986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973113344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829303.991994:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829303.991997:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829303.992000:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973113344:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829303.992029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829300/real 1787829303] req@ffff9c1a7895c000 x1874668973114112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829303.992048:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973114112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829303.992057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829303.992060:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829303.992064:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973114112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829304.503442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829304.503442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829304.759495:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829305.527460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829305.527460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829305.783476:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829306.039426:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829306.039437:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829306.039441:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829306.039453:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5756 00000100:00080000:3.0:1787829306.039463:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829306.039472:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829306.039506:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973114240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829306.039528:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829306.039532:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829306.039534:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829306.039538:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5756 00000100:00080000:3.0:1787829306.039542:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829306.039547:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829306.039557:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829306.039574:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973114240:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829306.039575:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973114368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829306.039594:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829306.039597:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829306.039600:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829306.039605:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5756 00000100:00080000:3.0:1787829306.039609:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829306.039613:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829306.039619:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829306.039633:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973114496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829306.039645:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829306.039650:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829306.039652:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829306.039655:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829306.039660:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5756 00000100:00080000:3.0:1787829306.039665:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829306.039671:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829306.039676:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829306.039707:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829306] req@ffff9c1a7895e580 x1874668973114240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829306.039709:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973114624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829306.039742:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829306.039746:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829306.039748:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829306.039755:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5756 00000100:00100000:0.0:1787829306.039756:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973114240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829306.039761:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829306.039770:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787829306.039776:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829306.039778:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829306.039786:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829306.039796:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973114240:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829306.039796:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973114752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829306.039813:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829306.039817:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829306.039819:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829306.039824:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5756 00000100:00080000:3.0:1787829306.039829:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829306.039831:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973114368:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829306.039842:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829306.039851:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829306.039866:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973114880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829306.039878:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829306.039879:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829306.039885:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829306.039888:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829306.039893:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5756 00000100:00080000:3.0:1787829306.039898:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829306.039899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829306] req@ffff9c1a7895c000 x1874668973114368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829306.039906:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829306.039919:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829306.039926:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973114368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829306.039936:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973115008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829306.039938:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829306.039948:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829306.039957:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973114368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829306.039971:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973114496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829306.039996:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829306.040010:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829306] req@ffff9c1a7895c780 x1874668973114496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829306.040023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973114496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829306.040032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829306.040036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829306.040040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973114496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829306.040047:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973114624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829306.040067:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829306.040079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829306] req@ffff9c1a7895c3c0 x1874668973114624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829306.040091:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973114624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829306.040099:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829306.040102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829306.040104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973114624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829306.040110:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973114752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829306.040129:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829306.040140:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829306] req@ffff9c1a7895e940 x1874668973114752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829306.040150:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973114752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829306.040158:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829306.040161:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829306.040164:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973114752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829306.040182:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973114880:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829306.040201:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829306.040211:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829306] req@ffff9c1a7895cf00 x1874668973114880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829306.040227:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973114880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829306.040235:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829306.040238:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829306.040240:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973114880:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829306.040246:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973115008:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829306.040264:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829306.040275:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829306/real 1787829306] req@ffff9c1a7895f480 x1874668973115008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829306.040287:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973115008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829306.040294:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829306.040297:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829306.040300:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973115008:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829306.551486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829306.551490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829306.807508:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829307.575415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829307.575418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829307.831630:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829308.599450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829308.599456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829308.855498:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829309.623388:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829309.623396:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829309.879534:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829310.647430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829310.647430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829310.903460:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829311.159452:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829311.159462:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829311.159467:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829311.159477:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5761 00000100:00080000:3.0:1787829311.159487:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829311.159496:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829311.159534:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973115136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829311.159563:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829311.159566:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829311.159568:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829311.159572:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5761 00000100:00080000:3.0:1787829311.159577:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829311.159581:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829311.159591:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829311.159605:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973115264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829311.159614:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973115136:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829311.159623:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829311.159628:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829311.159630:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829311.159635:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5761 00000100:00080000:3.0:1787829311.159640:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829311.159659:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829311.159666:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829311.159679:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973115392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829311.159688:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829311.159696:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829311.159701:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829311.159703:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829311.159709:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5761 00000100:00080000:3.0:1787829311.159713:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829311.159718:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829311.159724:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829311.159725:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff9c1a7895d2c0 x1874668973115136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829311.159741:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973115520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829311.159744:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973115136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.159772:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829311.159775:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829311.159776:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829311.159783:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829311.159784:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973115136:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829311.159789:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829311.159796:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5761 00000100:00080000:3.0:1787829311.159801:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829311.159806:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973115264:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829311.159808:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829311.159817:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829311.159831:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829311.159835:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973115648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829311.159849:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff9c1a7895cb40 x1874668973115264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829311.159850:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829311.159859:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829311.159864:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829311.159872:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5761 00000100:00080000:3.0:1787829311.159877:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829311.159878:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973115264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829311.159886:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787829311.159892:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829311.159896:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829311.159897:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829311.159903:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973115264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829311.159914:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973115776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829311.159919:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973115392:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829311.159933:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829311.159936:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829311.159939:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829311.159945:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5761 00000800:00000100:0.0:1787829311.159946:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829311.159954:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829311.159960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff9c1a7895da40 x1874668973115392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829311.159962:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829311.159973:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829311.159976:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973115392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.159989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829311.159991:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973115904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829311.159994:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829311.160002:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973115392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829311.160016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973115520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829311.160040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829311.160053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff9c1a7895e1c0 x1874668973115520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829311.160062:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973115520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.160068:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829311.160070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829311.160072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973115520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829311.160085:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973115648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829311.160098:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829311.160105:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff9c1a7895fc00 x1874668973115648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829311.160113:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973115648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.160118:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829311.160120:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829311.160123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973115648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829311.160126:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973115776:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829311.160137:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829311.160142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff9c1a7895f840 x1874668973115776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829311.160149:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973115776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.160154:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829311.160156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829311.160158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973115776:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829311.160161:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973115904:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829311.160174:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829311.160184:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829311/real 1787829311] req@ffff9c1a7895f0c0 x1874668973115904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829311.160196:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973115904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829311.160205:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829311.160208:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829311.160212:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973115904:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829311.671463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829311.671470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829311.927497:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829312.695427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829312.695430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829312.951503:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829313.719484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829313.719496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829313.975538:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829314.743453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829314.743453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829314.999539:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829315.767498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829315.767501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829316.023544:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829316.279482:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829316.279497:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829316.279503:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829316.279516:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5766 00000100:00080000:0.0:1787829316.279533:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829316.279544:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829316.279594:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973116032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829316.279646:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829316.279651:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829316.279654:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829316.279662:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5766 00000100:00080000:0.0:1787829316.279670:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829316.279679:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829316.279695:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829316.279695:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973116032:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829316.279727:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973116160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829316.279746:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829316.279750:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829316.279754:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829316.279758:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5766 00000100:00080000:0.0:1787829316.279764:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829316.279771:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829316.279777:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829316.279791:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973116288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829316.279804:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829316.279809:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829316.279811:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829316.279817:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5766 00000100:00080000:0.0:1787829316.279820:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829316.279825:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829316.279830:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829316.279838:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829316.279849:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973116416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829316.279859:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829316.279862:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829316.279864:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829316.279867:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5766 00000100:00080000:0.0:1787829316.279871:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829316.279875:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829316.279880:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829316.279891:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973116544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829316.279902:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829316.279906:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829316.279908:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829316.279912:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5766 00000100:00080000:0.0:1787829316.279916:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829316.279920:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829316.279925:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829316.279937:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973116672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829316.279949:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829316.279955:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829316.279958:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829316.279965:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5766 00000100:00080000:0.0:1787829316.279970:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829316.279976:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829316.279983:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829316.279999:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973116800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829316.280044:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff9c1a78860780 x1874668973116032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829316.280064:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973116032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829316.280077:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829316.280080:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829316.280083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973116032:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829316.280104:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973116160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829316.280124:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829316.280135:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff9c1a78861e00 x1874668973116160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829316.280148:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973116160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829316.280158:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829316.280162:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829316.280165:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973116160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829316.280173:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973116288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829316.280194:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829316.280203:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff9c1a788612c0 x1874668973116288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829316.280211:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973116288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829316.280216:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829316.280219:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829316.280221:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973116288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829316.280226:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973116416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829316.280242:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829316.280252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff9c1a78863c00 x1874668973116416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829316.280265:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973116416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829316.280274:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829316.280278:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829316.280281:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973116416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829316.280288:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973116544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829316.280306:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829316.280314:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff9c1a788630c0 x1874668973116544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829316.280322:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973116544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829316.280328:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829316.280330:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829316.280332:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973116544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829316.280336:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973116672:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829316.280378:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829316.280391:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff9c1a78862580 x1874668973116672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829316.280404:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973116672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829316.280411:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829316.280414:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829316.280417:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973116672:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829316.280422:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973116800:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829316.280435:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829316.280444:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829316/real 1787829316] req@ffff9c1a78861680 x1874668973116800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829316.280456:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973116800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829316.280465:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829316.280468:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829316.280472:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973116800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829316.791449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829316.791449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829317.047510:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829317.815470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829317.815470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829318.071502:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829318.839438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829318.839451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829319.095523:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829319.863430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829319.863430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829320.119544:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829320.887436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829320.887485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829321.143522:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829321.399440:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829321.399449:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829321.399453:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829321.399461:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5771 00000100:00080000:3.0:1787829321.399470:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829321.399475:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829321.399509:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973116928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829321.399532:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829321.399535:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829321.399536:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829321.399539:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5771 00000100:00080000:3.0:1787829321.399541:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829321.399544:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829321.399552:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829321.399562:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973117056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829321.399570:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829321.399586:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829321.399588:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973116928:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829321.399588:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829321.399595:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5771 00000100:00080000:3.0:1787829321.399601:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829321.399603:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829321.399607:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829321.399617:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973117184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829321.399626:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829321.399628:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829321.399629:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829321.399633:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5771 00000100:00080000:3.0:1787829321.399637:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829321.399639:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829321.399643:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829321.399649:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973117312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829321.399656:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829321.399658:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829321.399659:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829321.399664:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829321.399665:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5771 00000100:00080000:3.0:1787829321.399671:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829321.399673:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829321.399676:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829321.399684:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973117440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829321.399691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829321.399693:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829321.399694:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829321.399696:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5771 00000100:00080000:3.0:1787829321.399699:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829321.399702:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829321.399705:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829321.399713:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff9c1a7895de00 x1874668973116928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829321.399715:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973117568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829321.399746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829321.399750:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829321.399752:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829321.399755:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5771 00000100:00100000:0.0:1787829321.399758:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973116928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829321.399759:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829321.399767:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829321.399774:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829321.399776:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.399783:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829321.399784:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973117696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829321.399789:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973116928:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829321.399816:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973117056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829321.399836:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829321.399845:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff9c1a7895f0c0 x1874668973117056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.399854:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973117056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.399860:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.399862:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829321.399864:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973117056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829321.399868:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973117184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829321.399878:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829321.399884:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff9c1a7895f840 x1874668973117184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.399891:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973117184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.399901:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.399903:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829321.399905:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973117184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829321.399908:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973117312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829321.399917:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829321.399923:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff9c1a7895fc00 x1874668973117312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.399929:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973117312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.399934:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.399936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829321.399938:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973117312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829321.399941:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973117440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829321.399950:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829321.399956:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff9c1a7895e1c0 x1874668973117440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.399962:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973117440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.399967:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.399969:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829321.399972:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973117440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829321.399975:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973117568:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829321.399986:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829321.399992:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff9c1a7895da40 x1874668973117568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.399998:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973117568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.400004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.400005:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829321.400007:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973117568:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829321.400010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973117696:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829321.400029:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829321.400036:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829321/real 1787829321] req@ffff9c1a7895cb40 x1874668973117696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829321.400043:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973117696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829321.400049:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829321.400051:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829321.400053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973117696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829321.911490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829321.911493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829322.167592:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829322.935435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829322.935461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829323.191546:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829323.959449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829323.959449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829324.215527:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829324.983433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829324.983461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829325.239497:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829326.007438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829326.007469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829326.263566:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829326.519474:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829326.519482:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829326.519486:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829326.519494:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5776 00000100:00080000:1.0:1787829326.519504:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829326.519510:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829326.519555:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973117824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829326.519579:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829326.519582:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829326.519584:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829326.519587:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5776 00000100:00080000:1.0:1787829326.519590:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829326.519592:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829326.519601:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829326.519612:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973117952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829326.519626:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829326.519628:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829326.519631:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829326.519638:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973117824:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829326.519639:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5776 00000100:00080000:1.0:1787829326.519647:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829326.519656:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829326.519660:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829326.519671:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973118080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829326.519679:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829326.519682:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829326.519683:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829326.519685:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5776 00000100:00080000:1.0:1787829326.519687:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829326.519690:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829326.519694:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829326.519702:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973118208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829326.519708:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829326.519710:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829326.519711:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829326.519715:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5776 00000100:00080000:1.0:1787829326.519719:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829326.519722:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829326.519725:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829326.519733:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829326.519735:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973118336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829326.519744:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829326.519747:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829326.519749:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829326.519752:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5776 00000100:00080000:1.0:1787829326.519754:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829326.519757:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829326.519761:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829326.519771:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973118464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829326.519778:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829326.519780:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829326.519781:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829326.519787:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5776 00000100:00100000:0.0:1787829326.519789:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829326/real 1787829326] req@ffff9c1a78a7a580 x1874668973117824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829326.519791:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829326.519798:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829326.519805:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829326.519819:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973118592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829326.519836:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973117824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829326.519852:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829326.519857:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829326.519861:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973117824:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829326.519886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973117952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829326.519907:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829326.519917:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829326/real 1787829326] req@ffff9c1a78a78f00 x1874668973117952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829326.519926:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973117952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829326.519932:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829326.519934:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829326.519937:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973117952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829326.519941:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973118080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829326.519951:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829326.519957:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829326/real 1787829326] req@ffff9c1a78a7b480 x1874668973118080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829326.519965:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973118080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829326.519983:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829326.519986:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829326.519988:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973118080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829326.519993:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973118208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829326.520006:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829326.520013:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829326/real 1787829326] req@ffff9c1a78a79a40 x1874668973118208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829326.520021:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973118208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829326.520027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829326.520030:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829326.520032:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973118208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829326.520035:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973118336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829326.520046:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829326.520052:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829326/real 1787829326] req@ffff9c1a78a78000 x1874668973118336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829326.520059:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973118336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829326.520065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829326.520066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829326.520069:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973118336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829326.520073:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973118464:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829326.520084:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829326.520090:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829326/real 1787829326] req@ffff9c1a78a792c0 x1874668973118464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829326.520097:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973118464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829326.520103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829326.520105:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829326.520107:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973118464:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829326.520111:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973118592:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829326.520121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829326.520127:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829326/real 1787829326] req@ffff9c1a78a7b840 x1874668973118592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829326.520134:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973118592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829326.520140:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829326.520141:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829326.520143:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973118592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829327.031522:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829327.031522:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829327.287552:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829328.055400:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829328.055438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829328.311484:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829329.079398:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829329.079423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829329.335489:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829330.103436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829330.103437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829330.359489:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829331.127453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829331.127453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829331.383492:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829331.639437:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829331.639445:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829331.639449:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829331.639456:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5781 00000100:00080000:3.0:1787829331.639466:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829331.639472:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829331.639515:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973118720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829331.639536:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829331.639539:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829331.639540:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829331.639544:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5781 00000100:00080000:3.0:1787829331.639546:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829331.639549:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829331.639558:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829331.639566:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973118848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829331.639575:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829331.639578:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829331.639579:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829331.639582:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5781 00000100:00080000:3.0:1787829331.639584:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829331.639586:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829331.639589:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829331.639603:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973118976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829331.639616:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973118720:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829331.639620:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829331.639630:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829331.639631:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829331.639634:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5781 00000100:00080000:3.0:1787829331.639637:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829331.639639:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829331.639643:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829331.639657:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973119104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829331.639665:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829331.639668:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829331.639669:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829331.639672:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5781 00000100:00080000:3.0:1787829331.639679:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829331.639682:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829331.639686:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829331.639694:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973119232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829331.639700:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829331.639702:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829331.639703:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829331.639710:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829331.639711:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5781 00000100:00080000:3.0:1787829331.639717:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829331.639720:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829331.639724:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829331.639733:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973119360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829331.639740:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829331.639743:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829331.639744:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829331.639747:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5781 00000100:00080000:3.0:1787829331.639752:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829331.639756:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829331.639760:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829331.639765:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829331/real 1787829331] req@ffff9c1a7895d2c0 x1874668973118720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829331.639771:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973119488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829331.639832:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973118720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829331.639843:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829331.639847:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829331.639850:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973118720:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829331.639857:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973118848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829331.639876:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829331.639890:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829331/real 1787829331] req@ffff9c1a7895f480 x1874668973118848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829331.639904:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973118848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829331.639914:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829331.639918:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829331.639921:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973118848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829331.639951:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973118976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829331.639972:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829331.639984:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829331/real 1787829331] req@ffff9c1a7895cf00 x1874668973118976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829331.639998:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973118976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829331.640009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829331.640012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829331.640014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973118976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829331.640019:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973119104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829331.640035:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829331.640043:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829331/real 1787829331] req@ffff9c1a7895e940 x1874668973119104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829331.640050:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973119104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829331.640056:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829331.640058:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829331.640060:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973119104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829331.640064:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973119232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829331.640075:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829331.640081:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829331/real 1787829331] req@ffff9c1a7895c3c0 x1874668973119232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829331.640088:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973119232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829331.640093:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829331.640095:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829331.640098:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973119232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829331.640102:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973119360:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829331.640114:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829331.640119:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829331/real 1787829331] req@ffff9c1a7895c780 x1874668973119360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829331.640127:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973119360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829331.640132:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829331.640134:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829331.640136:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973119360:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829331.640139:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973119488:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829331.640149:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829331.640155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829331/real 1787829331] req@ffff9c1a7895c000 x1874668973119488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829331.640163:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973119488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829331.640168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829331.640170:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829331.640172:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973119488:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829332.151433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829332.151433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829332.407458:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829333.175457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829333.175457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829333.431550:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829334.199417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829334.199417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829334.455511:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829335.223451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829335.223451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829335.479497:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829336.247421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829336.247426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829336.503496:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829336.759450:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829336.759459:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829336.759463:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829336.759485:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5787 00000100:00080000:3.0:1787829336.759496:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829336.759504:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829336.759543:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973119616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829336.759569:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829336.759572:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829336.759573:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829336.759576:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5787 00000100:00080000:3.0:1787829336.759579:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829336.759582:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829336.759590:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829336.759602:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973119744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829336.759610:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973119616:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829336.759624:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829336.759627:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829336.759629:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829336.759633:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5787 00000100:00080000:3.0:1787829336.759635:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829336.759638:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829336.759642:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829336.759651:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973119872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829336.759659:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829336.759666:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829336.759667:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829336.759673:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829336.759673:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5787 00000100:00080000:3.0:1787829336.759678:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829336.759682:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829336.759685:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829336.759694:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973120000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829336.759700:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829336.759702:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829336.759704:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829336.759706:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5787 00000100:00080000:3.0:1787829336.759709:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829336.759711:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829336.759714:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829336.759722:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829336/real 1787829336] req@ffff9c1a7895e580 x1874668973119616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829336.759723:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973120128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829336.759732:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829336.759734:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829336.759740:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829336.759748:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5787 00000100:00080000:3.0:1787829336.759752:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829336.759756:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829336.759759:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829336.759764:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973119616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829336.759768:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973120256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829336.759776:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829336.759778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829336.759780:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829336.759787:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5787 00000100:00080000:3.0:1787829336.759791:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829336.759795:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829336.759799:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829336.759800:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829336.759806:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973120384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829336.759810:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829336.759820:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973119616:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829336.759849:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973119744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829336.759876:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829336.759891:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829336/real 1787829336] req@ffff9c1a7895c000 x1874668973119744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829336.759906:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973119744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829336.759916:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829336.759919:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829336.759923:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973119744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829336.759929:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973119872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829336.759946:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829336.759954:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829336/real 1787829336] req@ffff9c1a7895c780 x1874668973119872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829336.759962:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973119872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829336.759977:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829336.759980:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829336.759983:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973119872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829336.759998:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973120000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829336.760020:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829336.760030:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829336/real 1787829336] req@ffff9c1a7895c3c0 x1874668973120000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829336.760038:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973120000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829336.760050:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829336.760052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829336.760055:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973120000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829336.760059:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973120128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829336.760074:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829336.760084:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829336/real 1787829336] req@ffff9c1a7895e940 x1874668973120128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829336.760096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973120128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829336.760105:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829336.760108:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829336.760112:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973120128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829336.760119:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973120256:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829336.760137:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829336.760148:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829336/real 1787829336] req@ffff9c1a7895cf00 x1874668973120256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829336.760160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973120256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829336.760168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829336.760172:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829336.760175:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973120256:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829336.760182:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973120384:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829336.760202:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829336.760213:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829336/real 1787829336] req@ffff9c1a7895f480 x1874668973120384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829336.760226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973120384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829336.760234:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829336.760238:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829336.760241:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973120384:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829337.271459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829337.271460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829337.527501:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829338.295432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829338.295432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829338.551496:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829339.319458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829339.319459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829339.575443:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829340.343449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829340.343452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829340.599489:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829341.367458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829341.367458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829341.623499:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829341.879466:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829341.879475:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829341.879479:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829341.879486:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5792 00000100:00080000:3.0:1787829341.879496:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829341.879501:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829341.879536:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973120512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829341.879559:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829341.879562:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829341.879564:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829341.879567:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5792 00000100:00080000:3.0:1787829341.879569:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829341.879572:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829341.879579:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829341.879589:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973120640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829341.879597:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829341.879600:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829341.879601:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829341.879606:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5792 00000100:00100000:0.0:1787829341.879607:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973120512:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829341.879608:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829341.879614:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829341.879619:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829341.879626:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973120768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829341.879643:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829341.879647:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829341.879649:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829341.879653:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5792 00000100:00080000:3.0:1787829341.879657:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829341.879661:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829341.879667:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829341.879676:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829341.879681:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973120896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829341.879693:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829341.879697:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829341.879700:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829341.879704:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5792 00000100:00080000:3.0:1787829341.879708:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829341.879711:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829341/real 1787829341] req@ffff9c1a7895d2c0 x1874668973120512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829341.879716:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829341.879728:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829341.879734:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973120512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829341.879746:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973121024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829341.879761:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829341.879767:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829341.879768:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829341.879772:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973120512:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829341.879775:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829341.879778:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829341.879783:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5792 00000100:00080000:3.0:1787829341.879787:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829341.879790:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829341.879795:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973120640:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829341.879795:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829341.879810:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973121152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829341.879819:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829341.879824:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829341.879825:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829341.879829:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829341.879835:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5792 00000100:00080000:3.0:1787829341.879839:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829341.879844:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829341/real 1787829341] req@ffff9c1a7895cb40 x1874668973120640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829341.879845:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829341.879854:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829341.879868:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973120640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829341.879874:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973121280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829341.879890:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829341.879895:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829341.879899:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973120640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829341.879904:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973120768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829341.879924:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829341.879932:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829341/real 1787829341] req@ffff9c1a7895da40 x1874668973120768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829341.879941:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973120768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829341.879965:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829341.879969:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829341.879972:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973120768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829341.879979:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973120896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829341.880000:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829341.880009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829341/real 1787829341] req@ffff9c1a7895e1c0 x1874668973120896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829341.880017:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973120896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829341.880022:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829341.880025:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829341.880027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973120896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829341.880031:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973121024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829341.880042:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829341.880048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829341/real 1787829341] req@ffff9c1a7895fc00 x1874668973121024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829341.880055:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973121024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829341.880060:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829341.880062:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829341.880064:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973121024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829341.880076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973121152:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829341.880087:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829341.880093:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829341/real 1787829341] req@ffff9c1a7895f840 x1874668973121152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829341.880104:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973121152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829341.880113:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829341.880116:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829341.880119:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973121152:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829341.880126:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973121280:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829341.880145:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829341.880156:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829341/real 1787829341] req@ffff9c1a7895f0c0 x1874668973121280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829341.880164:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973121280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829341.880170:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829341.880172:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829341.880174:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973121280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829342.391414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829342.391418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829342.647495:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829343.415475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829343.415475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829343.671524:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829344.439439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829344.439440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829344.695516:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829345.463485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829345.463487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829345.719583:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829346.487441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829346.487465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829346.743528:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829346.999473:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829346.999488:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829346.999494:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829346.999508:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5797 00000100:00080000:2.0:1787829346.999523:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829346.999533:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829346.999583:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973121408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829346.999616:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829346.999620:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829346.999623:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829346.999628:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5797 00000100:00080000:2.0:1787829346.999633:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829346.999639:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829346.999651:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829346.999658:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973121408:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829346.999671:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973121536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829346.999685:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829346.999689:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829346.999692:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829346.999697:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5797 00000100:00080000:2.0:1787829346.999702:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829346.999707:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829346.999713:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829346.999728:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973121664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829346.999734:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829346.999754:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829346.999758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829346.999760:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829346.999766:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5797 00000100:00080000:2.0:1787829346.999771:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829346.999776:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829346.999783:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829346.999789:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829346/real 1787829346] req@ffff9c1a49d8f0c0 x1874668973121408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829346.999800:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973121792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829346.999808:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973121408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829346.999820:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829346.999823:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829346.999826:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829346.999829:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829346.999832:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829346.999834:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973121408:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829346.999841:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5797 00000100:00080000:2.0:1787829346.999847:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829346.999852:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829346.999859:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829346.999861:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973121536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829346.999880:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973121920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829346.999886:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829346.999899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829346/real 1787829346] req@ffff9c1a49d8e1c0 x1874668973121536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829346.999900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829346.999912:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829346.999917:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829346.999920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973121536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829346.999927:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5797 00000100:00080000:0.0:1787829346.999931:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829346.999934:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829346.999936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829346.999941:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973121536:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829346.999946:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829346.999950:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973121664:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787829346.999956:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829346.999967:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829346.999976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973122048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829346.999979:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829346/real 1787829346] req@ffff9c1a49d8c3c0 x1874668973121664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829347.000008:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973121664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829347.000011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829347.000022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829347.000027:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829347.000032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829347.000037:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5797 00000100:00080000:0.0:1787829347.000038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829347.000044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973121664:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829347.000047:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829347.000050:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973121792:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829347.000057:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829347.000065:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829347.000068:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829347.000077:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829346/real 1787829347] req@ffff9c1a49d8e580 x1874668973121792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829347.000087:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973122176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829347.000088:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973121792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829347.000118:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829347.000123:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829347.000126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973121792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829347.000141:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973121920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829347.000154:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829347.000160:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829346/real 1787829347] req@ffff9c1a49d8fc00 x1874668973121920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829347.000167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973121920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829347.000173:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829347.000174:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829347.000177:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973121920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829347.000181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973122048:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829347.000191:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829347.000197:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829346/real 1787829347] req@ffff9c1a49d8ed00 x1874668973122048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829347.000204:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973122048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829347.000209:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829347.000211:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829347.000213:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973122048:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829347.000216:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973122176:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829347.000225:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829347.000231:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829346/real 1787829347] req@ffff9c1a49d8d680 x1874668973122176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829347.000238:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973122176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829347.000244:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829347.000245:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829347.000247:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973122176:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829347.511440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829347.511444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829347.767571:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829348.535439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829348.535439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829348.791536:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829349.559445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829349.559445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829349.815526:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829350.583517:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829350.583518:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829350.839553:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829351.607425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829351.607477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829351.863571:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829352.119514:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829352.119524:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829352.119528:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829352.119537:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5802 00000100:00080000:3.0:1787829352.119547:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829352.119557:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829352.119591:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973122304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829352.119625:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829352.119629:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829352.119630:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829352.119634:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5802 00000100:00080000:3.0:1787829352.119637:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829352.119642:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829352.119658:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829352.119672:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973122304:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829352.119673:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973122432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829352.119693:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829352.119696:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829352.119699:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829352.119704:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5802 00000100:00080000:3.0:1787829352.119707:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829352.119709:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829352.119712:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829352.119721:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973122560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829352.119729:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829352.119732:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829352.119733:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829352.119736:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5802 00000800:00000100:2.0:1787829352.119762:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829352.119812:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829352.119818:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829352.119824:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829352.119834:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973122688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829352.119842:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829352.119844:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829352.119846:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829352.119855:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff9c1a7895de00 x1874668973122304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829352.119877:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973122304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829352.119888:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829352.119892:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829352.119896:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973122304:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829352.119925:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973122432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829352.119946:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829352.119959:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff9c1a7895f0c0 x1874668973122432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829352.119966:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5802 00000100:00080000:3.0:1787829352.119978:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829352.119984:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973122432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829352.120000:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829352.120004:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829352.120010:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973122432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829352.120018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973122560:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829352.120024:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829352.120033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829352.120049:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829352.120051:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973122816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829352.120068:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829352.120071:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829352.120074:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829352.120079:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5802 00000100:00100000:2.0:1787829352.120083:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff9c1a7895f840 x1874668973122560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829352.120085:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829352.120096:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829352.120111:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829352.120116:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973122560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829352.120124:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829352.120135:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829352.120137:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973122944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829352.120142:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973122560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829352.120148:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973122688:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829352.120173:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829352.120182:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829352.120182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829352.120189:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829352.120198:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff9c1a7895fc00 x1874668973122688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829352.120199:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5802 00000100:00080000:3.0:1787829352.120214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829352.120218:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973122688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829352.120229:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829352.120232:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829352.120241:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787829352.120243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829352.120244:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973122688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829352.120273:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973123072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829352.120274:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973122816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829352.120299:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829352.120316:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff9c1a7895e1c0 x1874668973122816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829352.120330:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973122816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829352.120341:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829352.120384:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829352.120389:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973122816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829352.120395:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973122944:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829352.120413:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829352.120434:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff9c1a7895da40 x1874668973122944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829352.120445:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973122944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829352.120451:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829352.120454:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829352.120456:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973122944:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829352.120466:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973123072:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829352.120478:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829352.120484:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829352/real 1787829352] req@ffff9c1a7895cb40 x1874668973123072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829352.120492:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973123072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829352.120500:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829352.120503:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829352.120507:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973123072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829352.631463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829352.631463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829352.887453:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829353.655429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829353.655429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829353.911466:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829354.679492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829354.679493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829354.935502:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829355.703403:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829355.703427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829355.959501:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829356.727455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829356.727455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829356.983529:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829357.239420:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.239427:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829357.239431:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.239439:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5807 00000100:00080000:3.0:1787829357.239448:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.239453:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829357.239487:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973123200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829357.239510:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.239513:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829357.239514:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.239518:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5807 00000100:00080000:3.0:1787829357.239520:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.239523:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829357.239531:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829357.239539:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973123328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829357.239547:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.239549:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829357.239551:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.239553:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5807 00000100:00080000:3.0:1787829357.239555:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.239557:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829357.239560:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829357.239566:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973123456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829357.239573:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973123200:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829357.239576:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.239583:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829357.239584:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.239595:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5807 00000100:00080000:3.0:1787829357.239599:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.239601:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829357.239605:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829357.239614:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973123584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829357.239624:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.239626:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829357.239628:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.239630:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5807 00000100:00080000:3.0:1787829357.239635:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.239637:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829357.239640:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829357.239647:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829357.239649:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973123712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829357.239657:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.239659:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829357.239660:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.239663:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5807 00000100:00080000:3.0:1787829357.239667:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.239669:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829357.239672:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829357.239680:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973123840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829357.239685:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829357.239687:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829357.239689:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829357.239695:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5807 00000100:00100000:2.0:1787829357.239696:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff9c1a7895d2c0 x1874668973123200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829357.239698:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829357.239703:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829357.239709:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829357.239719:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973123968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829357.239740:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973123200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829357.239753:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829357.239758:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829357.239762:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973123200:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829357.239770:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973123328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829357.239796:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829357.239810:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff9c1a7895f480 x1874668973123328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829357.239824:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973123328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829357.239835:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829357.239839:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829357.239842:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973123328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829357.239868:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973123456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829357.239888:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829357.239899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff9c1a7895cf00 x1874668973123456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829357.239918:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973123456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829357.239926:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829357.239929:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829357.239933:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973123456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829357.239939:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973123584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829357.239958:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829357.239969:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff9c1a7895e940 x1874668973123584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829357.239981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973123584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829357.239989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829357.239992:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829357.239996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973123584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829357.240002:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973123712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829357.240019:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829357.240030:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff9c1a7895c3c0 x1874668973123712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829357.240042:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973123712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829357.240050:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829357.240054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829357.240057:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973123712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829357.240062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973123840:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829357.240080:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829357.240090:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff9c1a7895c780 x1874668973123840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829357.240101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973123840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829357.240110:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829357.240113:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829357.240117:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973123840:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829357.240123:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973123968:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829357.240140:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829357.240150:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829357/real 1787829357] req@ffff9c1a7895c000 x1874668973123968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829357.240167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973123968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829357.240176:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829357.240179:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829357.240182:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973123968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829357.751478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829357.751510:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829358.007473:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829358.775452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829358.775452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829359.031541:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829359.799441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829359.799441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829360.055473:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829360.823451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829360.823451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829361.079470:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829361.847480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829361.847482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829362.103514:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829362.359452:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829362.359463:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829362.359468:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829362.359479:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5812 00000100:00080000:0.0:1787829362.359491:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829362.359498:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829362.359542:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973124096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829362.359571:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829362.359574:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829362.359576:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829362.359579:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5812 00000100:00080000:0.0:1787829362.359582:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829362.359585:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829362.359595:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829362.359608:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973124224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829362.359617:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829362.359620:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829362.359622:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829362.359627:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973124096:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829362.359628:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5812 00000100:00080000:0.0:1787829362.359632:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829362.359637:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829362.359647:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829362.359660:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973124352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829362.359669:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829362.359672:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829362.359674:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829362.359676:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5812 00000100:00080000:0.0:1787829362.359680:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829362.359682:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829362.359686:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829362.359693:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973124480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829362.359700:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829362.359703:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829362.359704:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829362.359706:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5812 00000100:00080000:0.0:1787829362.359708:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829362.359710:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829362.359714:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829362.359715:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829362.359724:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973124608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829362.359731:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829362.359734:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829362.359736:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829362.359739:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5812 00000100:00080000:0.0:1787829362.359743:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829362.359746:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829362.359749:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829362.359757:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973124736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829362.359763:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829362.359765:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829362.359767:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829362.359771:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff9c1a78860000 x1874668973124096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829362.359773:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5812 00000100:00080000:0.0:1787829362.359779:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829362.359786:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829362.359791:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829362.359803:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973124096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829362.359804:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973124864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829362.359839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829362.359844:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829362.359848:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973124096:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829362.359875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973124224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829362.359894:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829362.359905:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff9c1a78861680 x1874668973124224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829362.359914:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973124224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829362.359920:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829362.359923:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829362.359925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973124224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829362.359929:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973124352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829362.359940:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829362.359945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff9c1a78862580 x1874668973124352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829362.359953:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973124352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829362.359959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829362.359961:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829362.359962:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973124352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829362.359966:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973124480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829362.359985:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829362.359993:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff9c1a788630c0 x1874668973124480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829362.360001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973124480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829362.360008:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829362.360010:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829362.360012:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973124480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829362.360016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973124608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829362.360028:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829362.360034:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff9c1a78863c00 x1874668973124608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829362.360041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973124608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829362.360047:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829362.360048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829362.360050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973124608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829362.360054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973124736:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829362.360065:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829362.360072:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff9c1a788612c0 x1874668973124736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829362.360082:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973124736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829362.360091:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829362.360094:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829362.360099:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973124736:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829362.360105:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973124864:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829362.360121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829362.360129:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829362/real 1787829362] req@ffff9c1a78861e00 x1874668973124864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829362.360137:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973124864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829362.360143:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829362.360145:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829362.360147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973124864:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829362.871473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829362.871473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829363.127486:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829363.895427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829363.895427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829364.151555:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829364.919477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829364.919477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829365.175527:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829365.943464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829365.943468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829366.199522:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829366.967462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829366.967462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829367.223573:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829367.479438:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829367.479446:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829367.479450:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829367.479458:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5817 00000100:00080000:3.0:1787829367.479467:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829367.479472:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829367.479503:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973124992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829367.479526:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829367.479529:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829367.479530:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829367.479534:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5817 00000100:00080000:3.0:1787829367.479536:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829367.479539:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829367.479546:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829367.479555:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973125120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829367.479568:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829367.479570:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829367.479572:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829367.479574:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5817 00000100:00080000:3.0:1787829367.479576:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829367.479578:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829367.479581:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829367.479586:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973124992:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829367.479591:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973125248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829367.479603:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829367.479605:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829367.479606:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829367.479609:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5817 00000100:00080000:3.0:1787829367.479611:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829367.479613:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829367.479616:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829367.479625:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973125376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829367.479631:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829367.479633:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829367.479634:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829367.479636:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5817 00000100:00080000:3.0:1787829367.479639:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829367.479642:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829367.479645:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829367.479652:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973125504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829367.479658:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829367.479660:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829367.479662:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829367.479665:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5817 00000100:00080000:3.0:1787829367.479668:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829367.479671:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829367.479676:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973125120:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829367.479676:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829367.479690:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973125632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829367.479696:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829367.479698:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829367.479699:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829367.479703:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5817 00000100:00080000:3.0:1787829367.479708:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829367.479710:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829367.479714:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829367.479724:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973125760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829367.479729:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973125248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829367.479760:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973125376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829367.479777:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973125504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829367.479789:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973125632:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829367.479803:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973125760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829367.991411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829367.991415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829368.247489:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829369.015437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829369.015437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829369.271488:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829370.039484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829370.039484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829370.295486:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829370.551586:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787829370.551621:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829370.551636:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829370.551674:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829370.551684:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829370.551693:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829370.551701:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829370.551712:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787829370.551718:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787829370.551729:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829367/real 1787829370] req@ffff9c1a7895c780 x1874668973125248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.551760:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973125248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.551772:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.551776:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829370.551779:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973125248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829370.551785:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829367/real 1787829370] req@ffff9c1a7895c3c0 x1874668973125376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.551793:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973125376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.551799:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.551801:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829370.551803:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973125376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829370.551807:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829367/real 1787829370] req@ffff9c1a7895e940 x1874668973125504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.551814:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973125504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.551819:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.551821:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829370.551824:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973125504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829370.551828:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829367/real 1787829370] req@ffff9c1a7895cf00 x1874668973125632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.551835:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973125632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.551841:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.551844:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829370.551848:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973125632:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829370.551854:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829367/real 1787829370] req@ffff9c1a7895e580 x1874668973124992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.551867:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973124992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.551875:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.551879:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829370.551883:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973124992:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829370.551890:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829367/real 1787829370] req@ffff9c1a7895c000 x1874668973125120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.551902:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973125120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.551910:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.551914:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829370.551917:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973125120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829370.551946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829367/real 1787829370] req@ffff9c1a7895f480 x1874668973125760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829370.551958:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973125760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829370.551967:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829370.551970:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829370.551974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973125760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829371.063407:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829371.063414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829371.319511:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829372.087452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829372.087452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829372.343475:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829372.599424:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829372.599432:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829372.599436:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829372.599444:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5822 00000100:00080000:3.0:1787829372.599453:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829372.599459:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829372.599496:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973125888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829372.599518:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829372.599521:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829372.599523:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829372.599529:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5822 00000100:00080000:3.0:1787829372.599533:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829372.599537:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829372.599547:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829372.599561:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973126016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829372.599568:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973125888:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829372.599582:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829372.599586:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829372.599589:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829372.599594:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5822 00000100:00080000:3.0:1787829372.599598:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829372.599602:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829372.599606:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829372.599616:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973126144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829372.599624:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829372.599628:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1787829372.599629:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829372.599631:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829372.599637:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5822 00000100:00080000:3.0:1787829372.599642:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829372.599645:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829372.599648:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829372.599660:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973126272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829372.599670:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829372.599675:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829372/real 1787829372] req@ffff9c1a7895d2c0 x1874668973125888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829372.599676:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829372.599680:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829372.599688:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5822 00000100:00080000:3.0:1787829372.599695:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829372.599700:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829372.599706:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973125888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829372.599706:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829372.599723:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829372.599725:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973126400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829372.599733:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829372.599742:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973125888:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829372.599757:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829372.599761:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829372.599763:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829372.599768:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5822 00000100:00080000:3.0:1787829372.599773:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829372.599774:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973126016:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829372.599781:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829372.599788:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829372.599802:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973126528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829372.599812:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829372.599813:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829372.599819:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829372.599822:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829372.599827:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5822 00000100:00080000:3.0:1787829372.599831:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829372.599835:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829372/real 1787829372] req@ffff9c1a7895cb40 x1874668973126016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829372.599840:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829372.599848:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829372.599864:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973126656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829372.599871:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973126016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829372.599891:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829372.599895:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829372.599898:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973126016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829372.599906:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973126144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829372.599928:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829372.599941:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829372/real 1787829372] req@ffff9c1a7895da40 x1874668973126144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829372.599953:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973126144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829372.599960:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829372.599963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829372.599966:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973126144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829372.599972:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973126272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829372.599990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829372.600002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829372/real 1787829372] req@ffff9c1a7895e1c0 x1874668973126272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829372.600013:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973126272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829372.600026:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829372.600029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829372.600033:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973126272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829372.600038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973126400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829372.600058:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829372.600069:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829372/real 1787829372] req@ffff9c1a7895fc00 x1874668973126400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829372.600082:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973126400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829372.600091:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829372.600095:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829372.600105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973126400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829372.600127:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973126528:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829372.600148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829372.600159:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829372/real 1787829372] req@ffff9c1a7895f840 x1874668973126528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829372.600170:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973126528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829372.600178:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829372.600181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829372.600185:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973126528:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829372.600191:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973126656:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829372.600210:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829372.600343:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829372/real 1787829372] req@ffff9c1a7895f0c0 x1874668973126656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829372.600385:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973126656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829372.600394:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829372.600397:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829372.600401:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973126656:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829373.111436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829373.111436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829373.367484:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829374.135460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829374.135460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829374.391518:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829375.159406:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829375.159434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829375.415462:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829376.183446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829376.183446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829376.439563:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829377.207436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829377.207440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829377.463511:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829377.719468:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829377.719482:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829377.719489:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829377.719502:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5828 00000100:00080000:3.0:1787829377.719516:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829377.719524:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829377.719571:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973126784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829377.719608:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829377.719611:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829377.719613:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829377.719618:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5828 00000100:00080000:3.0:1787829377.719623:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829377.719628:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829377.719640:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829377.719659:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973126912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829377.719665:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973126784:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829377.719678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829377.719683:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829377.719686:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829377.719691:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5828 00000100:00080000:3.0:1787829377.719695:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829377.719701:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829377.719707:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829377.719722:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973127040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829377.719740:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829377.719741:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829377.719746:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829377.719749:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829377.719754:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5828 00000100:00080000:3.0:1787829377.719760:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829377.719765:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829377.719771:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829377.719785:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973127168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829377.719789:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829377/real 1787829377] req@ffff9c1a7895de00 x1874668973126784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829377.719817:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973126784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829377.719821:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829377.719833:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829377.719837:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829377.719838:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829377.719844:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829377.719849:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973126784:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829377.719849:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5828 00000100:00080000:3.0:1787829377.719859:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829377.719866:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829377.719872:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829377.719880:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973126912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829377.719888:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973127296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829377.719902:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829377.719904:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829377.719910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829377.719913:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829377.719918:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829377/real 1787829377] req@ffff9c1a7895f0c0 x1874668973126912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829377.719920:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5828 00000100:00080000:3.0:1787829377.719930:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829377.719934:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973126912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829377.719940:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829377.719949:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829377.719952:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829377.719954:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829377.719959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973126912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829377.719964:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973127040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829377.719971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973127424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829377.719980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829377.719990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829377.719991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829377/real 1787829377] req@ffff9c1a7895f840 x1874668973127040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829377.720001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829377.720006:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829377.720008:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973127040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829377.720017:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5828 00000100:00080000:2.0:1787829377.720019:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829377.720026:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829377.720026:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829377.720029:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973127040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829377.720036:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973127168:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829377.720046:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829377.720050:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829377.720055:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829377.720060:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829377/real 1787829377] req@ffff9c1a7895fc00 x1874668973127168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829377.720075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973127552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829377.720076:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973127168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829377.720107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829377.720112:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829377.720115:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973127168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829377.720128:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973127296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829377.720141:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829377.720149:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829377/real 1787829377] req@ffff9c1a7895e1c0 x1874668973127296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829377.720158:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973127296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829377.720164:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829377.720166:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829377.720168:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973127296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829377.720171:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973127424:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829377.720206:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829377.720215:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829377/real 1787829377] req@ffff9c1a7895da40 x1874668973127424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829377.720222:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973127424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829377.720233:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829377.720235:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829377.720237:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973127424:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829377.720242:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973127552:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829377.720253:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829377.720258:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829377/real 1787829377] req@ffff9c1a7895cb40 x1874668973127552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829377.720266:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973127552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829377.720271:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829377.720273:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829377.720275:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973127552:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829378.231465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829378.231465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829378.487508:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829379.255510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829379.255529:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829379.511504:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829380.279489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829380.279502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829380.535562:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829381.303432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829381.303483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829381.559471:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829382.327499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829382.327504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829382.583494:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829382.839460:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829382.839470:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829382.839475:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829382.839486:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5833 00000100:00080000:0.0:1787829382.839498:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829382.839505:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829382.839535:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973127680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829382.839561:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829382.839564:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829382.839566:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829382.839569:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5833 00000100:00080000:0.0:1787829382.839572:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829382.839574:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829382.839582:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829382.839590:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973127808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829382.839604:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829382.839606:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829382.839608:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829382.839610:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5833 00000100:00080000:0.0:1787829382.839612:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829382.839615:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829382.839618:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829382.839628:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973127936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829382.839636:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973127680:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829382.839639:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829382.839654:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829382.839656:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829382.839659:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5833 00000100:00080000:0.0:1787829382.839662:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829382.839664:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829382.839666:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829382.839680:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973128064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829382.839686:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829382.839689:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829382.839690:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829382.839692:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5833 00000100:00080000:0.0:1787829382.839695:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829382.839699:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829382.839702:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829382.839712:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973128192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829382.839719:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829382.839722:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829382.839724:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829382.839728:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5833 00000100:00080000:0.0:1787829382.839733:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829382.839734:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829382.839745:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829382.839753:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829382.839770:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973128320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829382.839781:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829382.839785:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829382.839787:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829382.839793:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5833 00000100:00080000:0.0:1787829382.839798:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829382.839801:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829382/real 1787829382] req@ffff9c1a78860780 x1874668973127680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829382.839803:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829382.839814:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829382.839831:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973128448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829382.839842:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973127680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829382.839867:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829382.839873:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829382.839880:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973127680:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829382.839890:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973127808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829382.839920:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829382.839939:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829382/real 1787829382] req@ffff9c1a78862940 x1874668973127808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829382.839948:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973127808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829382.839955:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829382.839957:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829382.839960:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973127808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829382.839988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973127936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829382.840002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829382.840009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829382/real 1787829382] req@ffff9c1a78861a40 x1874668973127936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829382.840017:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973127936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829382.840023:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829382.840025:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829382.840028:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973127936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829382.840032:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973128064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829382.840043:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829382.840049:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829382/real 1787829382] req@ffff9c1a78860f00 x1874668973128064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829382.840057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973128064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829382.840063:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829382.840065:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829382.840067:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973128064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829382.840071:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973128192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829382.840082:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829382.840087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829382/real 1787829382] req@ffff9c1a788621c0 x1874668973128192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829382.840095:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973128192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829382.840100:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829382.840102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829382.840104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973128192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829382.840108:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973128320:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829382.840120:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829382.840125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829382/real 1787829382] req@ffff9c1a78863840 x1874668973128320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829382.840138:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973128320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829382.840144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829382.840165:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829382.840167:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973128320:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829382.840172:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973128448:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829382.840192:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829382.840201:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829382/real 1787829382] req@ffff9c1a78862d00 x1874668973128448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829382.840209:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973128448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829382.840215:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829382.840217:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829382.840219:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973128448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829383.351423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829383.351448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829383.607572:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829384.375462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829384.375468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829384.631521:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829385.399411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829385.399451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829385.655579:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829386.423484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829386.423523:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829386.679613:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829387.447416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829387.447449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829387.703593:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829387.959456:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829387.959466:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829387.959470:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829387.959478:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5838 00000100:00080000:3.0:1787829387.959487:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829387.959494:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829387.959525:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973128576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829387.959553:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829387.959556:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829387.959557:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829387.959561:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5838 00000100:00080000:3.0:1787829387.959564:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829387.959567:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829387.959577:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829387.959585:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973128576:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829387.959598:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973128704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829387.959612:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829387.959616:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829387.959618:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829387.959624:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5838 00000100:00080000:3.0:1787829387.959630:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829387.959636:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829387.959643:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829387.959648:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829387.959662:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973128832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829387.959673:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829387.959675:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829387.959677:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829387.959681:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5838 00000100:00080000:3.0:1787829387.959685:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829387.959687:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829387/real 1787829387] req@ffff9c1a7895d2c0 x1874668973128576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829387.959691:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829387.959699:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829387.959714:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973128960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829387.959720:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973128576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829387.959735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829387.959741:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829387.959743:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829387.959744:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829387.959752:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829387.959756:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5838 00000100:00100000:0.0:1787829387.959762:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973128576:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829387.959764:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829387.959771:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829387.959777:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829387.959788:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973129088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829387.959796:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973128704:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829387.959798:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829387.959805:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829387.959807:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829387.959810:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5838 00000100:00080000:3.0:1787829387.959814:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829387.959816:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829387.959820:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829387.959828:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973129216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829387.959836:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829387.959850:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829387/real 1787829387] req@ffff9c1a7895f480 x1874668973128704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829387.959864:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973128704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829387.959870:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829387.959873:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829387.959875:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973128704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829387.959880:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973128832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829387.959893:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829387.959902:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829387/real 1787829387] req@ffff9c1a7895c000 x1874668973128832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829387.959924:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973128832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829387.959925:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829387.959934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829387.959936:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829387.959947:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829387.959950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829387.959952:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973128832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829387.959956:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973128960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829387.959969:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829387.959976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829387/real 1787829387] req@ffff9c1a7895e580 x1874668973128960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829387.959984:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973128960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829387.959989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829387.959991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829387.959993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973128960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829387.960004:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973129088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829387.960017:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829387.960024:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829387/real 1787829387] req@ffff9c1a7895cf00 x1874668973129088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829387.960033:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973129088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829387.960039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829387.960041:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829387.960044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973129088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829387.960049:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973129216:192.168.203.148@tcp:8:kworker.0 00000100:00080000:3.0:1787829387.960059:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5838 00000800:00000100:0.0:1787829387.960064:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829387.960065:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829387.960073:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829387.960081:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829387/real 1787829387] req@ffff9c1a7895e940 x1874668973129216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829387.960081:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829387.960094:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973129216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829387.960101:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829387.960105:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829387.960124:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973129216:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829387.960146:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973129344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829387.960200:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973129344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829387.960228:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829387.960240:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829387/real 1787829387] req@ffff9c1a7895c3c0 x1874668973129344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829387.960251:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973129344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829387.960258:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829387.960261:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829387.960263:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973129344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829388.471476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829388.471479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829388.727574:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829389.495451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829389.495482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829389.751551:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829390.519460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829390.519467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829390.775532:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829391.543491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829391.543494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829391.799553:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829392.567424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829392.567463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829392.823540:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829393.079457:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.079466:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829393.079471:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.079479:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5843 00000100:00080000:3.0:1787829393.079488:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829393.079494:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829393.079527:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973129472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829393.079559:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.079563:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829393.079565:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.079576:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5843 00000100:00080000:3.0:1787829393.079582:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829393.079586:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829393.079594:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829393.079606:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973129600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829393.079615:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.079619:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829393.079621:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.079625:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5843 00000100:00080000:3.0:1787829393.079629:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829393.079636:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973129472:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829393.079638:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829393.079651:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829393.079670:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973129728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829393.079690:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.079694:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829393.079697:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.079703:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5843 00000100:00080000:3.0:1787829393.079707:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829393.079712:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829393.079720:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829393.079734:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829393.079739:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973129856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829393.079749:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.079753:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829393.079756:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.079763:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5843 00000100:00080000:3.0:1787829393.079767:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829393.079773:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829393.079781:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829393.079789:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff9c1a7895c3c0 x1874668973129472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829393.079800:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973129984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829393.079812:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829393.079813:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973129472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829393.079827:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829393.079834:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.079842:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5843 00000100:00080000:0.0:1787829393.079843:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829393.079854:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829393.079855:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829393.079865:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829393.079868:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973129472:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787829393.079873:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829393.079886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973129600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829393.079893:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973130112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829393.079907:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829393.079911:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829393.079914:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829393.079920:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5843 00000800:00000100:0.0:1787829393.079929:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829393.079931:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829393.079937:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829393.079943:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829393.079951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff9c1a7895c780 x1874668973129600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829393.079959:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973130240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829393.079998:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973129600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.080008:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.080012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829393.080017:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973129600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829393.080051:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973129728:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829393.080066:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829393.080087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff9c1a7895e940 x1874668973129728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829393.080098:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973129728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.080103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.080106:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829393.080108:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973129728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829393.080113:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973129856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829393.080128:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829393.080136:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff9c1a7895cf00 x1874668973129856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829393.080144:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973129856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.080150:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.080152:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829393.080154:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973129856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829393.080158:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973129984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829393.080168:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829393.080173:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff9c1a7895e580 x1874668973129984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829393.080181:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973129984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.080186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.080188:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829393.080190:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973129984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829393.080194:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973130112:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829393.080205:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829393.080211:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff9c1a7895c000 x1874668973130112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829393.080218:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973130112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.080228:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.080230:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829393.080232:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973130112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829393.080236:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973130240:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829393.080246:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829393.080252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829393/real 1787829393] req@ffff9c1a7895f480 x1874668973130240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829393.080268:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973130240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829393.080274:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829393.080276:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829393.080278:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973130240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829393.591489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829393.591492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829393.847545:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829394.615429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829394.615449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829394.871548:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829395.639511:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829395.639512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829395.895585:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829396.663451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829396.663502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829396.919458:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829397.687480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829397.687487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829397.943584:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829398.199482:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.199492:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829398.199496:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.199508:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5848 00000100:00080000:3.0:1787829398.199520:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.199528:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829398.199574:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973130368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829398.199603:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.199606:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829398.199607:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.199615:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5848 00000100:00080000:3.0:1787829398.199618:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.199621:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829398.199631:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829398.199644:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973130496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829398.199655:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.199660:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829398.199662:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.199667:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5848 00000100:00100000:0.0:1787829398.199674:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973130368:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829398.199674:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.199682:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829398.199689:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829398.199703:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973130624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829398.199716:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.199721:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829398.199723:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.199727:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5848 00000100:00080000:3.0:1787829398.199732:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.199737:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829398.199743:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829398.199756:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973130752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829398.199766:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.199769:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829398.199771:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829398.199778:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829398.199778:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5848 00000100:00080000:3.0:1787829398.199786:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.199791:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829398.199797:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829398.199811:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973130880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829398.199822:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829398.199825:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829398.199828:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.199833:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5848 00000100:00100000:0.0:1787829398.199839:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff9c1a7895d2c0 x1874668973130368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829398.199840:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.199849:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829398.199860:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829398.199873:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973130368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829398.199878:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973131008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829398.199902:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.199913:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829398.199916:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829398.199922:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973130368:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829398.199924:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829398.199929:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829398.199936:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5848 00000100:00100000:0.0:1787829398.199939:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973130496:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829398.199943:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829398.199951:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829398.199957:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829398.199973:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973131136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829398.199977:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829398.199998:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff9c1a7895cb40 x1874668973130496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829398.200015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973130496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829398.200025:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.200029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829398.200033:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973130496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829398.200076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973130624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829398.200101:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829398.200114:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff9c1a7895da40 x1874668973130624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829398.200127:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973130624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829398.200134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.200137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829398.200139:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973130624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829398.200145:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973130752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829398.200168:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829398.200181:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff9c1a7895e1c0 x1874668973130752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829398.200195:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973130752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829398.200216:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.200219:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829398.200223:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973130752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829398.200230:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973130880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829398.200252:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829398.200265:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff9c1a7895fc00 x1874668973130880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829398.200278:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973130880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829398.200286:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.200290:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829398.200293:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973130880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829398.200312:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973131008:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829398.200333:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829398.200383:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff9c1a7895f840 x1874668973131008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829398.200399:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973131008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829398.200416:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.200420:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829398.200424:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973131008:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829398.200430:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973131136:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829398.200452:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829398.200465:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829398/real 1787829398] req@ffff9c1a7895f0c0 x1874668973131136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829398.200479:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973131136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829398.200488:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829398.200491:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829398.200495:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973131136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829398.711461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829398.711466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829398.967474:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829399.735480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829399.735481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829399.991553:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829400.759455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829400.759455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829401.015503:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829401.783453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829401.783457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829402.039609:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829402.807418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829402.807420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829403.063535:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829403.319439:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829403.319447:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829403.319450:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829403.319458:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5853 00000100:00080000:0.0:1787829403.319468:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829403.319473:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829403.319512:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973131264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829403.319532:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829403.319535:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829403.319537:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829403.319540:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5853 00000100:00080000:0.0:1787829403.319542:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829403.319545:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829403.319553:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829403.319566:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973131392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829403.319568:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973131264:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829403.319582:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829403.319585:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829403.319586:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829403.319589:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5853 00000100:00080000:0.0:1787829403.319591:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829403.319594:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829403.319597:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829403.319606:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973131520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829403.319613:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829403.319615:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829403.319617:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829403.319620:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5853 00000100:00080000:0.0:1787829403.319622:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829403.319624:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:3.0:1787829403.319629:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787829403.319630:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829403.319641:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973131648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829403.319648:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829403.319650:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829403.319651:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829403.319654:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5853 00000100:00080000:0.0:1787829403.319656:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829403.319659:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829403.319662:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829403.319666:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff9c1a78860b40 x1874668973131264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829403.319673:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973131776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829403.319683:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829403.319686:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973131264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829403.319687:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829403.319691:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829403.319699:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5853 00000100:00080000:0.0:1787829403.319703:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829403.319705:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829403.319707:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787829403.319711:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787829403.319712:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829403.319717:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973131264:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829403.319723:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973131904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829403.319730:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829403.319732:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829403.319734:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829403.319737:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5853 00000100:00100000:3.0:1787829403.319739:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973131392:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829403.319741:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829403.319748:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829403.319752:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829403.319761:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973132032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829403.319767:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829403.319783:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff9c1a78862d00 x1874668973131392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829403.319793:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973131392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829403.319799:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829403.319801:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829403.319803:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973131392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829403.319808:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973131520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829403.319818:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829403.319824:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff9c1a78863840 x1874668973131520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829403.319831:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973131520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829403.319837:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829403.319839:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829403.319841:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973131520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829403.319845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973131648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829403.319855:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829403.319862:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff9c1a788621c0 x1874668973131648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829403.319869:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973131648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829403.319874:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829403.319876:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829403.319878:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973131648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829403.319881:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973131776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829403.319891:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829403.319896:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff9c1a78860f00 x1874668973131776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829403.319904:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973131776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829403.319909:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829403.319910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829403.319912:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973131776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829403.319916:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973131904:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829403.319925:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829403.319930:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff9c1a78861a40 x1874668973131904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829403.319937:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973131904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829403.319943:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829403.319945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829403.319947:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973131904:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829403.319958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973132032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829403.319967:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829403.319972:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829403/real 1787829403] req@ffff9c1a78862940 x1874668973132032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829403.319979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973132032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829403.319985:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829403.319987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829403.319990:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973132032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829403.831444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829403.831450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829404.087485:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829404.855416:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829404.855417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829405.111562:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829405.879463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829405.879463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829406.135523:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829406.903414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829406.903429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829407.159522:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829407.927460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829407.927460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829408.183489:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829408.439464:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.439473:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829408.439477:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.439485:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5858 00000100:00080000:1.0:1787829408.439495:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829408.439501:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829408.439531:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973132160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829408.439555:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.439558:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829408.439559:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.439562:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5858 00000100:00080000:1.0:1787829408.439565:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829408.439567:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829408.439575:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829408.439584:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973132288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829408.439591:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.439593:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829408.439594:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.439596:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5858 00000100:00080000:1.0:1787829408.439599:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829408.439601:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829408.439604:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829408.439609:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973132160:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829408.439613:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973132416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829408.439622:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.439624:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829408.439625:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.439628:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5858 00000100:00080000:1.0:1787829408.439629:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829408.439632:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829408.439635:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829408.439643:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973132544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829408.439649:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.439651:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829408.439652:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.439654:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5858 00000100:00080000:1.0:1787829408.439657:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829408.439659:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829408.439662:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829408.439667:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829408.439671:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973132672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829408.439683:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.439691:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829408.439692:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.439696:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5858 00000100:00080000:1.0:1787829408.439699:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829408.439703:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff9c1a78a7bc00 x1874668973132160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829408.439704:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829408.439712:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829408.439724:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973132800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829408.439731:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973132160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829408.439734:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829408.439741:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829408.439744:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829408.439749:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5858 00000100:00080000:3.0:1787829408.439750:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829408.439751:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829408.439762:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787829408.439763:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829408.439769:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829408.439770:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973132160:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829408.439778:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973132928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829408.439781:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973132288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829408.439808:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829408.439817:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff9c1a78a78780 x1874668973132288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829408.439826:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973132288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829408.439832:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829408.439834:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829408.439836:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973132288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829408.439854:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973132416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829408.439865:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829408.439872:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff9c1a78a7a940 x1874668973132416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829408.439880:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973132416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829408.439889:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829408.439891:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829408.439893:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973132416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829408.439897:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973132544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829408.439906:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829408.439911:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff9c1a78a783c0 x1874668973132544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829408.439918:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973132544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829408.439924:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829408.439926:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829408.439927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973132544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829408.439931:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973132672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829408.439941:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829408.439947:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff9c1a78a7a1c0 x1874668973132672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829408.439961:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973132672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829408.439978:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829408.439981:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829408.439984:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973132672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829408.439988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973132800:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829408.440003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829408.440011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff9c1a78a7b0c0 x1874668973132800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829408.440024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973132800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829408.440030:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829408.440032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829408.440034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973132800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829408.440038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973132928:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829408.440049:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829408.440055:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829408/real 1787829408] req@ffff9c1a78a79e00 x1874668973132928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829408.440063:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973132928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829408.440067:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829408.440069:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829408.440071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973132928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829408.951429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829408.951430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829409.207489:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829409.975432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829409.975444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829410.231494:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829410.999446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829410.999446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829411.255435:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829412.023426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829412.023426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829412.279521:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829413.047447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829413.047447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829413.303521:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829413.559459:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829413.559469:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829413.559474:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829413.559484:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5863 00000100:00080000:3.0:1787829413.559495:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829413.559503:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829413.559540:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973133056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829413.559568:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829413.559572:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829413.559574:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829413.559578:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5863 00000100:00080000:3.0:1787829413.559583:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829413.559587:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829413.559597:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829413.559620:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973133056:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829413.559626:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973133184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829413.559641:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829413.559644:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829413.559647:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829413.559652:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5863 00000100:00080000:3.0:1787829413.559656:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829413.559661:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829413.559667:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829413.559680:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973133312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829413.559692:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829413.559695:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829413.559697:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829413.559702:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5863 00000100:00080000:3.0:1787829413.559707:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829413.559711:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829413.559716:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829413.559724:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829413.559738:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973133440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829413.559748:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829413.559752:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829413.559754:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829413.559760:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5863 00000100:00080000:3.0:1787829413.559765:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829413.559767:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829413/real 1787829413] req@ffff9c1a7895de00 x1874668973133056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829413.559771:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829413.559780:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829413.559795:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973133056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829413.559798:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973133568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829413.559820:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829413.559823:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829413.559828:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829413.559830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829413.559836:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973133056:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829413.559836:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829413.559845:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5863 00000100:00080000:3.0:1787829413.559850:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829413.559855:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829413.559860:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829413.559868:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973133184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829413.559874:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973133696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829413.559887:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829413.559891:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829413.559893:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829413.559898:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5863 00000800:00000100:0.0:1787829413.559903:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829413.559904:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829413.559911:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829413.559917:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829413.559925:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829413/real 1787829413] req@ffff9c1a7895f0c0 x1874668973133184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829413.559945:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973133824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829413.559970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973133184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829413.559996:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829413.560000:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829413.560004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973133184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829413.560010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973133312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829413.560032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829413.560044:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829413/real 1787829413] req@ffff9c1a7895f840 x1874668973133312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829413.560056:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973133312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829413.560065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829413.560069:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829413.560072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973133312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829413.560078:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973133440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829413.560097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829413.560108:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829413/real 1787829413] req@ffff9c1a7895fc00 x1874668973133440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829413.560119:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973133440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829413.560128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829413.560131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829413.560144:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973133440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829413.560150:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973133568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829413.560170:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829413.560182:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829413/real 1787829413] req@ffff9c1a7895e1c0 x1874668973133568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829413.560194:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973133568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829413.560202:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829413.560206:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829413.560209:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973133568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829413.560227:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973133696:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829413.560249:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829413.560260:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829413/real 1787829413] req@ffff9c1a7895da40 x1874668973133696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829413.560271:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973133696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829413.560279:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829413.560282:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829413.560285:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973133696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829413.560291:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973133824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829413.560308:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829413.560319:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829413/real 1787829413] req@ffff9c1a7895cb40 x1874668973133824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829413.560331:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973133824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829413.560339:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829413.560342:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829413.560374:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973133824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829414.071434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829414.071435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829414.327497:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829415.095447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829415.095447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829415.351525:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829416.119408:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829416.119416:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829416.375494:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829417.143459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829417.143614:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829417.399565:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829418.167444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829418.167467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829418.423484:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829418.679507:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829418.679516:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829418.679519:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829418.679527:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5869 00000100:00080000:2.0:1787829418.679536:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829418.679542:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829418.679581:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973133952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829418.679614:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829418.679617:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829418.679619:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829418.679622:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5869 00000100:00080000:2.0:1787829418.679625:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829418.679629:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829418.679636:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829418.679647:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973134080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829418.679656:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829418.679659:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829418.679661:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829418.679664:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5869 00000100:00080000:2.0:1787829418.679666:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829418.679668:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829418.679671:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829418.679678:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973134208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829418.679688:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829418.679691:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829418.679694:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829418.679697:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973133952:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829418.679698:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5869 00000100:00080000:2.0:1787829418.679702:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829418.679712:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829418.679717:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829418.679723:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973134336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829418.679728:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829418.679730:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829418.679731:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829418.679733:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5869 00000100:00080000:2.0:1787829418.679738:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829418.679740:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829418.679743:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829418.679749:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973134464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829418.679755:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829418.679756:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829418.679758:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829418.679760:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5869 00000100:00080000:2.0:1787829418.679765:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829418.679768:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829418.679772:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829418.679780:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973134592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829418.679786:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829418.679788:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829418.679790:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829418.679796:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5869 00000800:00000100:0.0:1787829418.679804:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829418.679806:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829418.679813:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829418.679818:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829418.679830:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973134720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829418.679860:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829418/real 1787829418] req@ffff9c1a49d8d680 x1874668973133952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829418.679878:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973133952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829418.679888:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829418.679892:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829418.679909:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973133952:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829418.679917:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973134080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829418.679939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829418.679949:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829418/real 1787829418] req@ffff9c1a49d8ed00 x1874668973134080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829418.679958:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973134080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829418.679969:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829418.679971:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829418.679974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973134080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829418.680001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973134208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829418.680014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829418.680020:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829418/real 1787829418] req@ffff9c1a49d8fc00 x1874668973134208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829418.680028:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973134208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829418.680034:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829418.680036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829418.680038:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973134208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829418.680042:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973134336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829418.680052:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829418.680058:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829418/real 1787829418] req@ffff9c1a49d8e580 x1874668973134336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829418.680065:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973134336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829418.680071:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829418.680073:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829418.680076:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973134336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829418.680080:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973134464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829418.680091:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829418.680098:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829418/real 1787829418] req@ffff9c1a49d8c3c0 x1874668973134464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829418.680110:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973134464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829418.680119:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829418.680122:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829418.680126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973134464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829418.680133:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973134592:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829418.680155:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829418.680164:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829418/real 1787829418] req@ffff9c1a49d8e1c0 x1874668973134592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829418.680173:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973134592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829418.680178:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829418.680181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829418.680183:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973134592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829418.680187:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973134720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829418.680199:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829418.680205:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829418/real 1787829418] req@ffff9c1a49d8f0c0 x1874668973134720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829418.680213:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973134720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829418.680219:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829418.680220:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829418.680222:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973134720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829419.191497:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829419.191499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829419.447547:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829420.215433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829420.215505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829420.471470:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829421.239432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829421.239432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829421.495463:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829422.263451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829422.263451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829422.519502:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829423.287455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829423.287455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829423.543534:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829423.799498:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829423.799507:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829423.799513:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829423.799524:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5874 00000100:00080000:3.0:1787829423.799535:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829423.799544:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829423.799582:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973134848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829423.799625:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829423.799630:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829423.799632:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829423.799638:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5874 00000100:00080000:3.0:1787829423.799644:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829423.799649:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829423.799666:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973134848:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787829423.799670:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829423.799692:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973134976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829423.799707:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829423.799711:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829423.799714:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829423.799718:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5874 00000100:00080000:3.0:1787829423.799724:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829423.799731:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829423.799731:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829423.799741:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829423.799756:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973135104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829423.799769:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829423.799773:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829423.799775:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829423.799778:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829423/real 1787829423] req@ffff9c1a7895d2c0 x1874668973134848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829423.799798:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973134848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829423.799809:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829423.799814:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829423.799818:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973134848:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829423.799841:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973134976:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829423.799853:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5874 00000100:00080000:3.0:1787829423.799873:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829423.799881:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829423.799885:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829423.799891:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829423.799904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829423/real 1787829423] req@ffff9c1a7895f480 x1874668973134976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829423.799911:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973135232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829423.799925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973134976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829423.799936:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829423.799940:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829423.799943:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973134976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829423.799949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973135104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829423.799967:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829423.799975:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829423/real 1787829423] req@ffff9c1a7895c000 x1874668973135104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829423.799983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973135104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829423.799989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829423.799991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829423.799993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973135104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829423.800005:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973135232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829423.800018:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829423.800025:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829423/real 1787829423] req@ffff9c1a7895e580 x1874668973135232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829423.800033:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973135232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829423.800039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829423.800041:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829423.800044:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973135232:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829423.800048:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829423.800053:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829423.800056:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829423.800062:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5874 00000100:00080000:3.0:1787829423.800068:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829423.800073:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829423.800121:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829423.800138:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973135360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829423.800159:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829423.800163:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829423.800165:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829423.800169:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5874 00000100:00080000:3.0:1787829423.800173:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829423.800177:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829423.800183:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829423.800195:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973135488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829423.800203:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829423.800207:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829423.800217:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829423.800223:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5874 00000100:00100000:2.0:1787829423.800227:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973135360:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829423.800229:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829423.800236:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829423.800242:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829423.800256:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973135616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829423.800264:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829423.800284:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829423/real 1787829423] req@ffff9c1a7895cf00 x1874668973135360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829423.800309:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973135360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829423.800318:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829423.800321:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829423.800324:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973135360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829423.800330:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973135488:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829423.800371:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829423.800386:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829423/real 1787829423] req@ffff9c1a7895e940 x1874668973135488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829423.800400:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973135488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829423.800410:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829423.800414:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829423.800419:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973135488:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829423.800437:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973135616:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829423.800456:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829423.800464:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829423/real 1787829423] req@ffff9c1a7895c780 x1874668973135616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829423.800471:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973135616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829423.800481:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829423.800483:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829423.800486:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973135616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829424.311445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829424.311445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829424.567508:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829425.335479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829425.335581:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829425.591518:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829426.359409:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829426.359410:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829426.615543:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829427.383469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829427.383469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829427.639579:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829428.407488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829428.407488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829428.663548:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829428.919470:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829428.919480:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829428.919485:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829428.919495:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5879 00000100:00080000:2.0:1787829428.919506:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829428.919512:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829428.919548:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973135744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829428.919579:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829428.919583:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829428.919585:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829428.919589:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5879 00000100:00080000:2.0:1787829428.919593:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829428.919597:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829428.919605:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829428.919616:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973135872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829428.919625:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829428.919628:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829428.919631:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829428.919635:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5879 00000100:00080000:2.0:1787829428.919641:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829428.919646:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829428.919652:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829428.919662:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973136000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829428.919676:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829428.919678:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973135744:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829428.919686:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829428.919690:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829428.919695:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5879 00000100:00080000:2.0:1787829428.919698:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829428.919704:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829428.919710:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829428.919724:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973136128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829428.919733:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829428.919737:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829428.919739:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829428.919743:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5879 00000100:00080000:2.0:1787829428.919751:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829428.919756:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829428.919763:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829428.919768:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829428.919779:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973136256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829428.919788:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829428.919792:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829428.919794:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829428.919798:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5879 00000100:00080000:2.0:1787829428.919801:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829428.919805:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829428.919810:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829428.919837:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829428/real 1787829428] req@ffff9c1a49d8cf00 x1874668973135744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829428.919841:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973136384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829428.919881:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829428.919896:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829428.919898:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829428.919905:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5879 00000100:00080000:2.0:1787829428.919909:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829428.919914:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973135744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829428.919918:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829428.919928:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829428.919936:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829428.919941:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829428.919946:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973136512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829428.919952:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973135744:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829428.919975:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973135872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829428.920006:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829428.920016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829428/real 1787829428] req@ffff9c1a49d8c000 x1874668973135872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829428.920025:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973135872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829428.920035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829428.920038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829428.920040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973135872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829428.920066:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973136000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829428.920078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829428.920085:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829428/real 1787829428] req@ffff9c1a49d8c780 x1874668973136000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829428.920092:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973136000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829428.920098:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829428.920099:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829428.920101:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973136000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829428.920106:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973136128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829428.920116:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829428.920123:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829428/real 1787829428] req@ffff9c1a49d8de00 x1874668973136128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829428.920130:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973136128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829428.920136:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829428.920137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829428.920140:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973136128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829428.920144:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973136256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829428.920155:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829428.920161:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829428/real 1787829428] req@ffff9c1a49d8da40 x1874668973136256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829428.920168:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973136256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829428.920173:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829428.920175:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829428.920177:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973136256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829428.920181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973136384:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829428.920192:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829428.920198:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829428/real 1787829428] req@ffff9c1a49d8cb40 x1874668973136384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829428.920206:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973136384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829428.920211:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829428.920212:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829428.920214:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973136384:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829428.920218:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973136512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829428.920228:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829428.920234:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829428/real 1787829428] req@ffff9c1a49d8f0c0 x1874668973136512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829428.920242:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973136512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829428.920247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829428.920249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829428.920251:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973136512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829429.431438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829429.431438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829429.687514:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829430.455414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829430.455414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829430.711490:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829431.479427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829431.479427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829431.735516:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829432.503432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829432.503432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829432.759445:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829433.527414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829433.527414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829433.783538:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829434.039474:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829434.039483:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829434.039488:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829434.039498:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5884 00000100:00080000:1.0:1787829434.039510:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829434.039519:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829434.039564:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973136640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829434.039589:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829434.039592:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829434.039593:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829434.039597:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5884 00000100:00080000:1.0:1787829434.039600:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829434.039603:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829434.039611:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829434.039622:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973136768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829434.039630:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829434.039633:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973136640:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829434.039634:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829434.039638:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829434.039643:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5884 00000100:00080000:1.0:1787829434.039645:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829434.039648:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829434.039651:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829434.039657:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973136896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829434.039665:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829434.039667:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829434.039669:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829434.039672:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5884 00000100:00080000:1.0:1787829434.039690:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829434.039693:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829434.039697:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829434.039704:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973137024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829434.039711:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829434.039713:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829434.039714:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829434.039718:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5884 00000100:00080000:1.0:1787829434.039720:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829434.039723:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829434.039727:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829434.039737:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973136768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829434.039738:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973137152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829434.039752:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829434.039755:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829434.039756:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829434.039759:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5884 00000100:00080000:1.0:1787829434.039762:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829434.039765:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829434.039774:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829434.039776:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973136896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829434.039803:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973137280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829434.039811:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829434.039814:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829434.039816:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829434.039821:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5884 00000100:00100000:0.0:1787829434.039822:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973137024:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829434.039825:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829434.039837:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829434.039842:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829434.039853:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973137408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829434.039870:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973137152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829434.039898:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973137280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829434.039912:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973137408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829434.551451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829434.551451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829434.807515:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829435.575449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829435.575449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829435.831522:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829436.599493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829436.599493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829436.855578:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829437.111614:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:3.0:1787829437.111665:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829437.111685:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829437.111736:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829437.111749:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829437.111761:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829437.111789:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829437.111801:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787829437.111812:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829437] req@ffff9c1a78a7b0c0 x1874668973136768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787829437.111813:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787829437.111846:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973136768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829437.111862:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829437.111869:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829437.111874:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973136768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829437.111883:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829437] req@ffff9c1a78a7a1c0 x1874668973136896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829437.111897:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973136896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829437.111905:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829437.111916:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829437.111920:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973136896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829437.111928:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829437] req@ffff9c1a78a783c0 x1874668973137024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829437.111943:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973137024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829437.111953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829437.111957:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829437.111961:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973137024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829437.111969:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829437] req@ffff9c1a78a79e00 x1874668973136640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829437.111983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973136640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829437.111992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829437.111997:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829437.112000:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973136640:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829437.112032:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829437] req@ffff9c1a78a7a940 x1874668973137152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829437.112046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973137152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829437.112053:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829437.112057:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829437.112060:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973137152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829437.112068:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829437] req@ffff9c1a78a78780 x1874668973137280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829437.112079:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973137280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829437.112086:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829437.112090:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829437.112093:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973137280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829437.112100:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829434/real 1787829437] req@ffff9c1a78a7bc00 x1874668973137408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829437.112111:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973137408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829437.112119:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829437.112123:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829437.112126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973137408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829437.623439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829437.623439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829437.879467:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829438.647442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829438.647486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829438.903537:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829439.159491:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829439.159503:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829439.159510:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829439.159524:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5889 00000100:00080000:1.0:1787829439.159538:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829439.159547:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829439.159593:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973137536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829439.159627:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829439.159630:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829439.159632:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829439.159635:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5889 00000100:00080000:1.0:1787829439.159638:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829439.159641:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829439.159651:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829439.159661:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973137664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829439.159670:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829439.159673:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829439.159674:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829439.159677:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5889 00000100:00080000:1.0:1787829439.159678:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829439.159681:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829439.159684:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829439.159691:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973137792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829439.159699:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973137536:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829439.159703:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829439.159710:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829439.159712:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829439.159717:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5889 00000100:00080000:1.0:1787829439.159720:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829439.159723:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829439.159726:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829439.159735:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973137920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829439.159746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829439.159749:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829439.159751:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829439.159766:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5889 00000100:00080000:1.0:1787829439.159769:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829439.159772:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829439.159777:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829439.159786:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973138048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829439.159794:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829439.159796:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829439.159798:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829439.159800:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5889 00000100:00080000:1.0:1787829439.159803:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829439.159806:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829439.159811:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829439.159812:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829439.159824:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973138176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829439.159833:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829439.159836:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829439.159838:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829439.159843:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5889 00000100:00080000:1.0:1787829439.159847:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829439.159851:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829439.159857:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829439.159872:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973138304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829439.159876:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829439] req@ffff9c1a78a7b840 x1874668973137536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829439.159915:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973137536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829439.159930:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829439.159936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829439.159942:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973137536:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829439.159963:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973137664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829439.159991:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829439.160006:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829439] req@ffff9c1a78a792c0 x1874668973137664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829439.160019:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973137664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829439.160028:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829439.160032:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829439.160036:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973137664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829439.160068:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973137792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829439.160091:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829439.160103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829439] req@ffff9c1a78a78000 x1874668973137792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829439.160116:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973137792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829439.160125:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829439.160128:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829439.160132:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973137792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829439.160140:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973137920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829439.160162:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829439.160175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829439] req@ffff9c1a78a79a40 x1874668973137920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829439.160188:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973137920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829439.160198:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829439.160202:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829439.160205:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973137920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829439.160212:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973138048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829439.160231:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829439.160243:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829439] req@ffff9c1a78a7b480 x1874668973138048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829439.160264:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973138048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829439.160273:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829439.160276:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829439.160279:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973138048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829439.160285:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973138176:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829439.160309:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829439.160321:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829439] req@ffff9c1a78a78f00 x1874668973138176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829439.160342:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973138176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829439.160409:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829439.160413:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829439.160417:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973138176:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829439.160424:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973138304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829439.160448:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829439.160459:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829439/real 1787829439] req@ffff9c1a78a7a580 x1874668973138304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829439.160471:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973138304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829439.160486:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829439.160490:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829439.160493:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973138304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829439.671436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829439.671436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829439.927459:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829440.695444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829440.695444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829440.951573:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829441.719438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829441.719466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829441.975516:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829442.743462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829442.743462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829442.999580:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829443.767446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829443.767446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829444.023568:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829444.279467:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829444.279482:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829444.279488:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829444.279503:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5894 00000100:00080000:1.0:1787829444.279518:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829444.279527:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829444.279582:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973138432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829444.279618:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829444.279623:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829444.279626:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829444.279630:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5894 00000100:00080000:1.0:1787829444.279635:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829444.279640:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829444.279652:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829444.279667:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973138560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829444.279680:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829444.279683:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829444.279686:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829444.279690:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5894 00000100:00080000:1.0:1787829444.279694:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829444.279699:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829444.279704:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829444.279716:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973138688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829444.279748:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829444.279752:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829444.279754:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829444.279759:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5894 00000100:00080000:1.0:1787829444.279764:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829444.279770:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829444.279777:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829444.279791:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973138816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829444.279802:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829444.279805:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829444.279808:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829444.279812:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5894 00000100:00080000:1.0:1787829444.279816:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829444.279821:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829444.279826:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829444.279842:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973138944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829444.279853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829444.279857:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829444.279859:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829444.279862:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973138432:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829444.279865:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5894 00000100:00080000:1.0:1787829444.279872:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829444.279878:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829444.279884:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829444.279898:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973139072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829444.279908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829444.279911:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829444.279914:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829444.279918:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5894 00000100:00080000:1.0:1787829444.279922:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829444.279926:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829444.279931:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829444.279945:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973139200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829444.279996:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829444.280052:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff9c1a78a7a580 x1874668973138432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829444.280075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973138432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829444.280088:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829444.280093:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829444.280098:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973138432:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829444.280107:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973138560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829444.280145:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829444.280175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff9c1a78a78f00 x1874668973138560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829444.280185:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973138560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829444.280192:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829444.280194:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829444.280197:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973138560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829444.280202:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973138688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829444.280224:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829444.280232:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff9c1a78a7b480 x1874668973138688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829444.280241:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973138688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829444.280247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829444.280249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829444.280252:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973138688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829444.280255:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973138816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829444.280268:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829444.280274:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff9c1a78a79a40 x1874668973138816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829444.280282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973138816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829444.280287:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829444.280296:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829444.280298:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973138816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829444.280325:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973138944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829444.280338:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829444.280373:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff9c1a78a78000 x1874668973138944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829444.280384:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973138944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829444.280390:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829444.280392:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829444.280395:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973138944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829444.280399:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973139072:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829444.280414:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829444.280421:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff9c1a78a792c0 x1874668973139072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829444.280429:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973139072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829444.280435:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829444.280437:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829444.280439:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973139072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829444.280442:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973139200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829444.280452:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829444.280457:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829444/real 1787829444] req@ffff9c1a78a7b840 x1874668973139200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829444.280464:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973139200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829444.280474:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829444.280477:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829444.280479:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973139200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829444.791445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829444.791448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829445.047590:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829445.815421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829445.815423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829446.071548:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829446.839451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829446.839451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829447.095485:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829447.863440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829447.863445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829448.119511:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829448.887434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829448.887434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829449.143544:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829449.399507:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.399517:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829449.399522:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.399534:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5899 00000100:00080000:0.0:1787829449.399546:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.399553:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829449.399592:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973139328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829449.399625:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.399629:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829449.399631:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.399636:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5899 00000100:00080000:0.0:1787829449.399640:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.399645:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829449.399654:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829449.399669:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973139456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829449.399675:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973139328:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829449.399705:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.399709:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829449.399711:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.399717:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5899 00000100:00080000:0.0:1787829449.399722:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.399726:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829449.399732:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829449.399746:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973139584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829449.399747:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829449.399762:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.399766:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829449.399768:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.399773:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5899 00000100:00080000:0.0:1787829449.399778:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.399785:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829449.399785:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff9c1a78860780 x1874668973139328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829449.399795:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829449.399804:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973139328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829449.399812:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973139712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829449.399815:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829449.399823:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829449.399829:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973139328:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829449.399835:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.399840:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829449.399842:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.399847:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5899 00000100:00080000:0.0:1787829449.399853:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829449.399855:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973139456:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829449.399861:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829449.399869:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829449.399879:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829449.399886:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973139840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829449.399896:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff9c1a78861e00 x1874668973139456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829449.399900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.399909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829449.399913:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.399920:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5899 00000100:00100000:2.0:1787829449.399924:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973139456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829449.399927:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.399937:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829449.399941:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829449.399945:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829449.399947:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829449.399954:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973139456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829449.399960:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973139584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829449.399965:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973139968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829449.399977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829449.399981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787829449.399983:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829449.399985:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829449.399993:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5899 00000100:00100000:2.0:1787829449.399997:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff9c1a788612c0 x1874668973139584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829449.400000:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829449.400010:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829449.400015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973139584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829449.400019:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829449.400023:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829449.400028:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829449.400030:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973139584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829449.400037:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973140096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829449.400038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973139712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829449.400058:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829449.400067:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff9c1a78863c00 x1874668973139712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829449.400076:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973139712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829449.400082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829449.400085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829449.400087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973139712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829449.400100:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973139840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829449.400111:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829449.400117:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff9c1a788630c0 x1874668973139840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829449.400129:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973139840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829449.400144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829449.400148:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829449.400151:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973139840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829449.400157:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973139968:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829449.400186:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829449.400206:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff9c1a78862580 x1874668973139968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829449.400217:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973139968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829449.400225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829449.400228:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829449.400231:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973139968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829449.400238:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973140096:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829449.400258:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829449.400270:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829449/real 1787829449] req@ffff9c1a78861680 x1874668973140096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829449.400282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973140096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829449.400291:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829449.400294:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829449.400298:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973140096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829449.911450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829449.911450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829450.167459:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829450.935465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829450.935465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829451.191490:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829451.959470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829451.959470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829452.215555:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829452.983417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829452.983429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829453.239494:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829454.007445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829454.007445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829454.263505:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829454.519406:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829454.519413:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829454.519417:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829454.519425:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5904 00000100:00080000:0.0:1787829454.519434:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829454.519440:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829454.519472:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973140224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829454.519498:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829454.519500:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829454.519502:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829454.519505:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5904 00000100:00080000:0.0:1787829454.519507:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829454.519509:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829454.519516:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829454.519526:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973140352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829454.519533:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829454.519536:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829454.519537:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829454.519539:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5904 00000100:00080000:0.0:1787829454.519541:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829454.519544:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829454.519547:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829454.519551:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973140224:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829454.519556:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973140480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829454.519570:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829454.519573:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829454.519574:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829454.519576:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5904 00000100:00080000:0.0:1787829454.519579:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829454.519581:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829454.519584:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829454.519592:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973140608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829454.519598:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829454.519600:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829454.519601:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829454.519603:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5904 00000100:00080000:0.0:1787829454.519605:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829454.519607:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829454.519612:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787829454.519613:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829454.519622:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973140736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829454.519629:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829454.519631:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829454.519632:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829454.519635:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5904 00000100:00080000:0.0:1787829454.519637:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829454.519640:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829454.519643:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829454.519670:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973140864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829454.519675:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829454/real 1787829454] req@ffff9c1a78860000 x1874668973140224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829454.519679:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829454.519688:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829454.519691:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829454.519698:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5904 00000100:00080000:0.0:1787829454.519701:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829454.519704:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829454.519708:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829454.519712:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973140224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829454.519718:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973140992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829454.519742:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829454.519747:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829454.519752:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973140224:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829454.519759:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973140352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829454.519779:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829454.519788:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829454/real 1787829454] req@ffff9c1a78861680 x1874668973140352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829454.519797:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973140352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829454.519802:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829454.519804:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829454.519807:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973140352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829454.519825:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973140480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829454.519838:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829454.519845:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829454/real 1787829454] req@ffff9c1a78862580 x1874668973140480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829454.519852:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973140480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829454.519857:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829454.519859:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829454.519861:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973140480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829454.519865:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973140608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829454.519874:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829454.519879:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829454/real 1787829454] req@ffff9c1a788630c0 x1874668973140608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829454.519886:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973140608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829454.519892:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829454.519893:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829454.519895:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973140608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829454.519899:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973140736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829454.519908:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829454.519913:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829454/real 1787829454] req@ffff9c1a78863c00 x1874668973140736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829454.519925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973140736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829454.519930:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829454.519931:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829454.519933:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973140736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829454.519936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973140864:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829454.519946:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829454.519951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829454/real 1787829454] req@ffff9c1a788612c0 x1874668973140864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829454.519958:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973140864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829454.519963:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829454.519965:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829454.519967:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973140864:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829454.519970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973140992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829454.519979:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829454.519984:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829454/real 1787829454] req@ffff9c1a78861e00 x1874668973140992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829454.519992:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973140992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829454.520000:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829454.520003:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829454.520005:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973140992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829455.031428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829455.031428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829455.287484:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829456.055437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829456.055437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829456.311538:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829457.079451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829457.079454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829457.335549:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829458.103489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829458.103496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829458.359581:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829459.127422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829459.127423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829459.383536:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829459.639454:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829459.639469:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829459.639475:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829459.639487:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5909 00000100:00080000:0.0:1787829459.639499:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829459.639508:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829459.639548:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973141120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829459.639577:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829459.639583:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829459.639585:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829459.639589:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5909 00000100:00080000:0.0:1787829459.639593:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829459.639597:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829459.639607:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829459.639630:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973141120:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829459.639633:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973141248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829459.639646:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829459.639649:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829459.639651:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829459.639655:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5909 00000100:00080000:0.0:1787829459.639658:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829459.639667:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829459.639672:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829459.639683:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973141376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829459.639692:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829459.639695:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829459.639697:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829459.639699:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5909 00000100:00080000:0.0:1787829459.639702:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829459.639705:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829459.639709:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829459.639715:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973141504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829459.639722:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829459.639725:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829459.639726:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829459.639728:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5909 00000100:00080000:0.0:1787829459.639729:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829459.639732:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829459.639734:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829459.639740:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973141632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829459.639745:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829459.639747:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829459.639748:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829459.639754:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5909 00000100:00080000:0.0:1787829459.639756:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829459.639759:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829459.639762:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829459.639769:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973141760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829459.639775:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829459.639777:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829459.639779:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829459.639781:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5909 00000100:00080000:0.0:1787829459.639783:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829459.639786:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829459.639788:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829459.639794:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973141888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829459.640943:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829459.640983:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829459/real 1787829459] req@ffff9c1a78860780 x1874668973141120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829459.640999:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973141120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829459.641007:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829459.641010:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829459.641014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973141120:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829459.641038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973141248:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829459.641059:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829459.641073:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829459/real 1787829459] req@ffff9c1a78862940 x1874668973141248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829459.641086:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973141248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829459.641096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829459.641100:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829459.641104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973141248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829459.641111:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973141376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829459.641130:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829459.641139:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829459/real 1787829459] req@ffff9c1a78861a40 x1874668973141376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829459.641147:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973141376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829459.641152:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829459.641154:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829459.641156:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973141376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829459.641160:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973141504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829459.641171:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829459.641177:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829459/real 1787829459] req@ffff9c1a78860f00 x1874668973141504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829459.641184:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973141504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829459.641190:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829459.641191:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829459.641193:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973141504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829459.641197:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973141632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829459.641207:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829459.641212:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829459/real 1787829459] req@ffff9c1a788621c0 x1874668973141632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829459.641219:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973141632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829459.641225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829459.641227:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829459.641229:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973141632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829459.641233:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973141760:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829459.641243:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829459.641248:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829459/real 1787829459] req@ffff9c1a78863840 x1874668973141760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829459.641255:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973141760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829459.641260:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829459.641263:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829459.641267:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973141760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829459.641274:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973141888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829459.641294:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829459.641305:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829459/real 1787829459] req@ffff9c1a78862d00 x1874668973141888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829459.641318:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973141888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829459.641324:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829459.641326:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829459.641329:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973141888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829460.151501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829460.151501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829460.407616:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829461.175456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829461.175456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829461.431521:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829462.199488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829462.199488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829462.455573:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829463.223431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829463.223432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829463.479482:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829464.247487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829464.247487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829464.503501:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829464.759503:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829464.759517:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829464.759522:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829464.759537:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5915 00000100:00080000:0.0:1787829464.759552:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829464.759561:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829464.759623:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973142016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829464.759657:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829464.759661:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829464.759663:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829464.759668:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5915 00000100:00080000:0.0:1787829464.759672:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829464.759676:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829464.759689:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829464.759710:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973142144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829464.759710:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973142016:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829464.759735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829464.759739:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829464.759741:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829464.759746:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5915 00000100:00080000:0.0:1787829464.759750:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829464.759755:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829464.759760:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829464.759773:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973142272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829464.759787:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829464.759798:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829464.759803:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829464.759805:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829464.759811:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5915 00000100:00080000:0.0:1787829464.759815:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829464.759820:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829464.759826:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829464.759833:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829464/real 1787829464] req@ffff9c1a78860b40 x1874668973142016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829464.759842:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973142400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829464.759854:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973142016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829464.759855:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829464.759871:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829464.759875:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829464.759877:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829464.759882:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5915 00000100:00080000:1.0:1787829464.759886:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829464.759890:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829464.759892:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973142016:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829464.759900:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829464.759906:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829464.759918:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973142528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829464.759921:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973142144:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829464.759932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829464.759937:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829464.759939:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829464.759945:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5915 00000100:00080000:0.0:1787829464.759950:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829464.759954:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829464.759959:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829464.759967:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829464.759969:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829464/real 1787829464] req@ffff9c1a78862d00 x1874668973142144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829464.759982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973142656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829464.759988:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973142144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829464.760001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829464.760005:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829464.760007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829464.760012:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829464.760015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829464.760021:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5915 00000100:00100000:1.0:1787829464.760025:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973142144:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829464.760028:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829464.760037:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829464.760041:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973142272:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787829464.760045:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829464.760063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973142784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829464.760073:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829464.760088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829464/real 1787829464] req@ffff9c1a78863840 x1874668973142272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829464.760107:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973142272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829464.760116:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829464.760120:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829464.760123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973142272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829464.760130:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973142400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829464.760151:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829464.760164:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829464/real 1787829464] req@ffff9c1a788621c0 x1874668973142400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829464.760175:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973142400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829464.760184:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829464.760187:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829464.760190:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973142400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829464.760207:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973142528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829464.760229:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829464.760241:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829464/real 1787829464] req@ffff9c1a78860f00 x1874668973142528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829464.760254:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973142528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829464.760263:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829464.760266:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829464.760270:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973142528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829464.760277:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973142656:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829464.760299:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829464.760312:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829464/real 1787829464] req@ffff9c1a78861a40 x1874668973142656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829464.760325:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973142656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829464.760334:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829464.760338:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829464.760341:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973142656:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829464.760377:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973142784:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829464.760399:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829464.760412:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829464/real 1787829464] req@ffff9c1a78862940 x1874668973142784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829464.760432:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973142784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829464.760441:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829464.760444:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829464.760448:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973142784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829465.271425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829465.271425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829465.527514:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829466.295455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829466.295455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829466.551547:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829467.319451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829467.319451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829467.575491:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829468.343421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829468.343426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829468.599469:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829469.367472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829469.367472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829469.623481:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829469.879464:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829469.879474:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829469.879479:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829469.879489:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5920 00000100:00080000:2.0:1787829469.879500:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829469.879508:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829469.879557:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973142912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829469.879584:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829469.879588:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829469.879590:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829469.879601:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5920 00000100:00080000:2.0:1787829469.879606:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829469.879611:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829469.879621:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829469.879628:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973142912:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829469.879637:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973143040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829469.879658:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829469.879662:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829469.879664:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829469.879669:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5920 00000100:00080000:2.0:1787829469.879673:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829469.879677:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829469.879683:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829469.879694:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829469.879697:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973143168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829469.879710:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829469.879714:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829469.879717:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829469.879721:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5920 00000100:00080000:2.0:1787829469.879725:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829469.879731:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829469.879737:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829469.879740:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829469/real 1787829469] req@ffff9c1a49d8e1c0 x1874668973142912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829469.879752:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973143296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829469.879759:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973142912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829469.879766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829469.879771:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829469.879774:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829469.879783:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5920 00000100:00080000:1.0:1787829469.879791:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829469.879791:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829469.879798:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787829469.879801:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:2.0:1787829469.879807:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829469.879809:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973142912:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829469.879823:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973143424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829469.879833:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829469.879836:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829469.879838:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829469.879843:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5920 00000100:00100000:1.0:1787829469.879848:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973143040:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829469.879849:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829469.879857:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829469.879865:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829469.879881:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973143552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829469.879887:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829469.879900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829469.879905:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829469/real 1787829469] req@ffff9c1a49d8c3c0 x1874668973143040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829469.879906:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829469.879911:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829469.879921:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5920 00000100:00100000:1.0:1787829469.879925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973143040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829469.879927:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829469.879937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829469.879938:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787829469.879944:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000008:00100000:2.0:1787829469.879948:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829469.879949:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973143040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829469.879959:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973143168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829469.879969:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973143680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829469.879978:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829469.879988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829469/real 1787829469] req@ffff9c1a49d8e580 x1874668973143168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829469.880003:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973143168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829469.880010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829469.880012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829469.880015:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973143168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829469.880020:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973143296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829469.880031:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829469.880037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829469/real 1787829469] req@ffff9c1a49d8fc00 x1874668973143296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829469.880055:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973143296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829469.880065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829469.880068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829469.880071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973143296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829469.880078:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973143424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829469.880113:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829469.880125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829469/real 1787829469] req@ffff9c1a49d8ed00 x1874668973143424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829469.880139:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973143424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829469.880148:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829469.880151:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829469.880154:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973143424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829469.880174:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973143552:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829469.880195:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829469.880206:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829469/real 1787829469] req@ffff9c1a49d8d680 x1874668973143552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829469.880215:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973143552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829469.880224:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829469.880226:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829469.880229:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973143552:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829469.880232:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973143680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829469.880244:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829469.880250:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829469/real 1787829469] req@ffff9c1a49d8e1c0 x1874668973143680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829469.880257:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973143680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829469.880262:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829469.880263:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829469.880265:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973143680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829470.391431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829470.391434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829470.647497:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829471.415495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829471.415495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829471.671485:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829472.439418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829472.439441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829472.695510:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829473.463477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829473.463477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829473.719549:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829474.487411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829474.487437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829474.743559:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829474.999470:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829474.999481:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829474.999486:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829474.999495:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5925 00000100:00080000:2.0:1787829474.999518:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829474.999525:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829474.999561:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973143808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829474.999589:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829474.999593:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829474.999595:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829474.999600:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5925 00000100:00080000:2.0:1787829474.999603:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829474.999608:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829474.999617:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829474.999628:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973143936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829474.999638:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829474.999641:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829474.999643:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829474.999646:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5925 00000100:00080000:2.0:1787829474.999649:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829474.999652:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829474.999657:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829474.999668:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973144064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829474.999680:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829474.999684:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829474.999686:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829474.999690:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5925 00000100:00080000:2.0:1787829474.999694:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829474.999698:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829474.999703:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829474.999712:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973144192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829474.999721:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829474.999725:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829474.999728:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829474.999735:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973143808:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829474.999735:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5925 00000100:00080000:2.0:1787829474.999742:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829474.999752:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829474.999758:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829474.999772:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973144320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829474.999782:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829474.999786:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829474.999788:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829474.999793:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5925 00000100:00080000:2.0:1787829474.999797:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829474.999800:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829474.999805:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829474.999816:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973144448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829474.999825:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829474.999828:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829474.999831:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829474.999835:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5925 00000100:00080000:2.0:1787829474.999848:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829474.999853:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787829474.999862:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787829474.999863:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829474.999879:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973144576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829474.999973:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829474/real 1787829474] req@ffff9c1a49d8f0c0 x1874668973143808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829474.999993:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973143808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829475.000003:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829475.000007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829475.000010:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973143808:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829475.000023:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973143936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829475.000051:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829475.000061:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829474/real 1787829475] req@ffff9c1a49d8cb40 x1874668973143936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829475.000070:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973143936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829475.000076:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829475.000079:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829475.000081:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973143936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829475.000085:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973144064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829475.000097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829475.000103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829474/real 1787829475] req@ffff9c1a49d8da40 x1874668973144064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829475.000111:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973144064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829475.000122:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829475.000124:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829475.000127:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973144064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829475.000155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973144192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829475.000166:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829475.000173:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829474/real 1787829475] req@ffff9c1a49d8de00 x1874668973144192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829475.000195:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973144192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829475.000247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829475.000252:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829475.000257:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973144192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829475.000264:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973144320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829475.000289:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829475.000302:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829474/real 1787829475] req@ffff9c1a49d8c780 x1874668973144320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829475.000311:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973144320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829475.000317:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829475.000319:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829475.000322:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973144320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829475.000327:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973144448:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829475.000418:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829475.000427:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829474/real 1787829475] req@ffff9c1a49d8c000 x1874668973144448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829475.000435:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973144448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829475.000441:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829475.000443:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829475.000445:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973144448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829475.000450:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973144576:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829475.000462:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829475.000468:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829475/real 1787829475] req@ffff9c1a49d8cf00 x1874668973144576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829475.000484:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973144576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829475.000490:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829475.000492:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829475.000494:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973144576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829475.511441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829475.511441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829475.767474:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829476.535436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829476.535443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829476.791472:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829477.559435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829477.559463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829477.815532:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829478.583441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829478.583441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829478.839540:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829479.607456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829479.607456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829479.863539:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829480.119514:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829480.119525:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829480.119529:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829480.119541:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5930 00000100:00080000:0.0:1787829480.119552:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829480.119560:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829480.119600:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973144704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829480.119631:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829480.119636:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829480.119638:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829480.119643:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5930 00000100:00080000:0.0:1787829480.119647:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829480.119652:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829480.119663:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829480.119678:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973144832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829480.119685:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973144704:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829480.119698:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829480.119702:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829480.119704:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829480.119709:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5930 00000100:00080000:0.0:1787829480.119714:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829480.119719:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829480.119726:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829480.119739:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973144960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829480.119744:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829480.119758:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829480.119763:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829480.119765:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829480.119770:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5930 00000100:00080000:0.0:1787829480.119776:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829480.119782:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829480.119785:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff9c1a78860780 x1874668973144704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829480.119789:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829480.119810:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973145088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829480.119813:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973144704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829480.119837:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829480.119839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829480.119845:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829480.119846:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829480.119849:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829480.119853:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973144704:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829480.119858:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5930 00000100:00080000:0.0:1787829480.119864:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829480.119869:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829480.119874:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829480.119882:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973144832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829480.119889:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973145216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829480.119903:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829480.119907:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829480.119910:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829480.119916:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829480.119920:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5930 00000100:00080000:0.0:1787829480.119926:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829480.119931:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829480.119935:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff9c1a78861e00 x1874668973144832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829480.119940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829480.119959:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973145344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829480.119960:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973144832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829480.119985:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829480.119988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829480.119990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829480.119996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829480.119999:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973144832:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829480.120000:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829480.120010:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5930 00000100:00080000:0.0:1787829480.120016:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829480.120020:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829480.120027:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829480.120030:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973144960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829480.120042:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973145472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829480.120062:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829480.120088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff9c1a788612c0 x1874668973144960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829480.120100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973144960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829480.120106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829480.120108:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829480.120110:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973144960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829480.120125:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973145088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829480.120147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829480.120159:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff9c1a78863c00 x1874668973145088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829480.120170:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973145088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829480.120178:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829480.120181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829480.120193:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973145088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829480.120211:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973145216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829480.120232:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829480.120245:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff9c1a788630c0 x1874668973145216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829480.120267:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973145216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829480.120276:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829480.120279:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829480.120282:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973145216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829480.120289:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973145344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829480.120325:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829480.120341:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff9c1a78862580 x1874668973145344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829480.120380:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973145344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829480.120391:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829480.120395:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829480.120398:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973145344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829480.120407:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973145472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829480.120424:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829480.120435:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829480/real 1787829480] req@ffff9c1a78861680 x1874668973145472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829480.120446:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973145472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829480.120486:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829480.120490:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829480.120493:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973145472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829480.631439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829480.631508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829480.887511:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829481.655444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829481.655445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829481.911525:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829482.679424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829482.679480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829482.935467:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829483.703489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829483.703489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829483.959486:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829484.727410:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829484.727449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829484.983559:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829485.239497:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.239517:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829485.239522:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.239533:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5935 00000100:00080000:0.0:1787829485.239547:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.239555:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829485.239606:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973145600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829485.239635:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.239638:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829485.239639:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.239643:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5935 00000100:00080000:0.0:1787829485.239646:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.239651:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829485.239663:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829485.239680:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973145728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829485.239687:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973145600:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829485.239695:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.239700:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829485.239702:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.239707:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5935 00000100:00080000:0.0:1787829485.239710:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.239713:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829485.239717:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829485.239726:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973145856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829485.239735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.239738:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829485.239739:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.239742:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5935 00000100:00080000:0.0:1787829485.239748:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.239751:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829485.239754:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829485.239761:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973145984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829485.239767:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829485.239772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.239775:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829485.239777:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.239780:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5935 00000100:00080000:0.0:1787829485.239782:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.239785:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829485.239788:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829485.239794:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973146112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829485.239799:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.239802:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829485.239803:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.239807:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5935 00000100:00080000:0.0:1787829485.239810:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829485.239814:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff9c1a78860000 x1874668973145600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829485.239817:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829485.239828:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829485.239836:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973145600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829485.239845:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973146240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829485.239847:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.239854:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829485.239860:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973145600:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829485.239863:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829485.239868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829485.239871:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829485.239876:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5935 00000100:00080000:0.0:1787829485.239880:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829485.239884:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829485.239890:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829485.239893:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973145728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829485.239905:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973146368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829485.239915:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829485.239927:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff9c1a78861680 x1874668973145728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829485.239937:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973145728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829485.239949:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.239952:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829485.239954:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973145728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829485.239958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973145856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829485.239970:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829485.239976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff9c1a78862580 x1874668973145856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829485.239983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973145856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829485.239987:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.239990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829485.239992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973145856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829485.239995:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973145984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829485.240004:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829485.240009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff9c1a788630c0 x1874668973145984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829485.240015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973145984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829485.240021:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.240022:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829485.240023:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973145984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829485.240027:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973146112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829485.240036:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829485.240041:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff9c1a78863c00 x1874668973146112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829485.240048:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973146112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829485.240053:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.240054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829485.240056:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973146112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829485.240060:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973146240:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829485.240078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829485.240087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff9c1a788612c0 x1874668973146240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829485.240099:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973146240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829485.240107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.240110:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829485.240113:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973146240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829485.240127:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973146368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829485.240141:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829485.240153:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829485/real 1787829485] req@ffff9c1a78861e00 x1874668973146368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829485.240166:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973146368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829485.240171:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829485.240173:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829485.240175:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973146368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829485.751461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829485.751462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829486.007526:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829486.775431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829486.775483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829487.031547:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829487.799471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829487.799471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829488.055523:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829488.823496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829488.823513:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829489.079503:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829489.847430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829489.847467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829490.103521:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829490.359512:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829490.359522:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829490.359526:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829490.359537:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5940 00000100:00080000:1.0:1787829490.359548:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829490.359555:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829490.359591:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973146496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829490.359624:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829490.359628:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829490.359630:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829490.359635:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5940 00000100:00080000:1.0:1787829490.359640:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829490.359645:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829490.359656:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829490.359670:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973146624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829490.359680:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973146496:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829490.359698:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829490.359703:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829490.359706:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829490.359711:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5940 00000100:00080000:1.0:1787829490.359715:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829490.359720:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829490.359727:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829490.359741:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973146752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829490.359755:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829490.359760:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829490.359762:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829490.359769:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829490.359775:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5940 00000100:00080000:1.0:1787829490.359783:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829490.359791:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829490.359797:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829490.359810:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973146880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829490.359822:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829490.359827:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff9c1a78a7bc00 x1874668973146496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829490.359829:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829490.359837:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829490.359853:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5940 00000100:00100000:2.0:1787829490.359859:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973146496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829490.359862:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829490.359876:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829490.359880:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829490.359885:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829490.359886:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829490.359891:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973146496:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829490.359907:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973147008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829490.359922:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829490.359926:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829490.359931:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829490.359931:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973146624:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829490.359943:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5940 00000100:00080000:1.0:1787829490.359948:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829490.359953:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829490.359960:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829490.359974:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829490.359980:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973147136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829490.359995:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff9c1a78a78780 x1874668973146624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829490.360003:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829490.360014:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829490.360014:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973146624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829490.360020:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829490.360039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829490.360042:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5940 00000100:00080000:2.0:1787829490.360046:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829490.360053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973146624:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829490.360054:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829490.360072:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973146752:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829490.360073:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829490.360089:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829490.360101:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829490.360112:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973147264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829490.360114:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff9c1a78a7a940 x1874668973146752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829490.360172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973146752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829490.360184:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829490.360188:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829490.360192:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973146752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829490.360212:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973146880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829490.360233:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829490.360245:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff9c1a78a79e00 x1874668973146880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829490.360256:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973146880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829490.360262:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829490.360264:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829490.360266:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973146880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829490.360271:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973147008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829490.360284:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829490.360290:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff9c1a78a783c0 x1874668973147008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829490.360297:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973147008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829490.360306:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829490.360308:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829490.360310:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973147008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829490.360323:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973147136:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829490.360334:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829490.360339:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff9c1a78a7a1c0 x1874668973147136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829490.360369:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973147136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829490.360374:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829490.360376:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829490.360378:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973147136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829490.360382:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973147264:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829490.360394:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829490.360414:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829490/real 1787829490] req@ffff9c1a78a7b0c0 x1874668973147264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829490.360422:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973147264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829490.360427:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829490.360429:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829490.360432:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973147264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829490.871500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829490.871501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829491.127657:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829491.895449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829491.895456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829492.151539:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829492.919498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829492.919498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829493.175527:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829493.943408:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829493.943444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829494.199547:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829494.967476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829494.967523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829495.223603:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829495.479475:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829495.479484:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829495.479487:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829495.479496:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5945 00000100:00080000:1.0:1787829495.479508:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829495.479516:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829495.479558:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973147392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829495.479589:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829495.479594:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829495.479596:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829495.479602:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5945 00000100:00080000:1.0:1787829495.479607:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829495.479613:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829495.479624:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829495.479640:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973147392:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829495.479642:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973147520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829495.479663:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829495.479668:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829495.479671:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829495.479677:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5945 00000100:00080000:1.0:1787829495.479682:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829495.479686:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829495.479691:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829495.479703:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973147648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829495.479713:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829495.479718:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829495.479720:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829495.479727:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829495.479732:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5945 00000100:00080000:1.0:1787829495.479741:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829495.479749:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829495.479758:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829495.479773:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973147776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829495.479789:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829495.479791:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829495/real 1787829495] req@ffff9c1a78a7b0c0 x1874668973147392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829495.479799:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829495.479803:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829495.479818:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5945 00000100:00080000:1.0:1787829495.479823:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829495.479830:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829495.479838:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829495.479839:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973147392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829495.479857:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829495.479859:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973147904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829495.479863:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829495.479876:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973147392:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829495.479893:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829495.479896:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829495.479898:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829495.479905:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5945 00000100:00080000:1.0:1787829495.479910:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829495.479910:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973147520:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829495.479921:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829495.479928:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829495.479943:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829495.479949:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973148032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829495.479958:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829495/real 1787829495] req@ffff9c1a78a7a1c0 x1874668973147520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829495.479984:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829495.479986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973147520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829495.479999:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829495.480005:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829495.480006:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829495.480011:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787829495.480021:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5945 00000100:00100000:2.0:1787829495.480021:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973147520:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829495.480036:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829495.480037:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973147648:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829495.480048:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829495.480059:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829495.480076:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973148160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829495.480083:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829495.480110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829495/real 1787829495] req@ffff9c1a78a783c0 x1874668973147648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829495.480126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973147648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829495.480133:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829495.480138:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829495.480142:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973147648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829495.480148:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973147776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829495.480183:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829495.480207:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829495/real 1787829495] req@ffff9c1a78a79e00 x1874668973147776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829495.480224:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973147776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829495.480233:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829495.480237:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829495.480241:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973147776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829495.480248:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973147904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829495.480267:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829495.480276:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829495/real 1787829495] req@ffff9c1a78a7a940 x1874668973147904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829495.480287:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973147904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829495.480293:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829495.480295:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829495.480297:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973147904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829495.480310:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973148032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829495.480323:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829495.480329:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829495/real 1787829495] req@ffff9c1a78a78780 x1874668973148032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829495.480335:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973148032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829495.480340:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829495.480343:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829495.480412:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973148032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829495.480420:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973148160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829495.480440:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829495.480451:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829495/real 1787829495] req@ffff9c1a78a7bc00 x1874668973148160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829495.480462:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973148160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829495.480475:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829495.480478:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829495.480481:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973148160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829495.991471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829495.991477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829496.247523:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829497.015458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829497.015473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829497.271511:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829498.039440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829498.039473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829498.295505:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829499.063459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829499.063495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829499.319596:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829500.087433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829500.087445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829500.343523:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829500.599461:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829500.599472:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829500.599478:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829500.599490:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5950 00000100:00080000:1.0:1787829500.599502:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829500.599509:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829500.599553:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973148288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829500.599575:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829500.599578:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829500.599579:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829500.599582:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5950 00000100:00080000:1.0:1787829500.599584:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829500.599586:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829500.599597:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829500.599606:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973148416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829500.599613:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829500.599616:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829500.599617:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829500.599620:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5950 00000100:00080000:1.0:1787829500.599623:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829500.599626:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973148288:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829500.599627:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829500.599631:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829500.599639:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973148544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829500.599651:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829500.599653:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829500.599654:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829500.599656:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5950 00000100:00080000:1.0:1787829500.599658:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829500.599660:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829500.599663:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829500.599671:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973148672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829500.599675:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829500.599677:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829500.599678:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829500.599680:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5950 00000100:00080000:1.0:1787829500.599682:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829500.599685:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829500.599687:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829500.599693:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973148800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829500.599698:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829500.599700:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829500.599702:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829500.599704:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5950 00000100:00080000:1.0:1787829500.599707:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829500.599710:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829500.599713:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829500.599720:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973148928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829500.599725:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829500.599728:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829500.599729:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829500.599732:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5950 00000100:00080000:1.0:1787829500.599734:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829500.599736:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829500.599740:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829500.599747:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973149056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829500.599769:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973148416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829500.599806:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973148544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829500.599829:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973148672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829500.599844:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973148800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829500.599856:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973148928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829500.599871:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973149056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829501.111441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829501.111441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829501.367572:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829502.135455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829502.135455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829502.391571:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829503.159455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829503.159456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829503.415520:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829503.671619:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787829503.671663:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829503.671681:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829503.671728:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829503.671738:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829503.671746:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829503.671754:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829503.671761:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829503.671769:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787829503.672012:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829500/real 1787829503] req@ffff9c1a78a792c0 x1874668973148416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.672038:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973148416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.672051:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.672057:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829503.672062:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973148416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829503.672070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829500/real 1787829503] req@ffff9c1a78a78000 x1874668973148544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.672084:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973148544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.672093:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.672102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829503.672107:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973148544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829503.672115:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829500/real 1787829503] req@ffff9c1a78a79a40 x1874668973148672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.672128:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973148672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.672136:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.672139:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829503.672142:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973148672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829503.672148:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829500/real 1787829503] req@ffff9c1a78a7b480 x1874668973148800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.672158:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973148800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.672165:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.672168:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829503.672171:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973148800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829503.672176:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829500/real 1787829503] req@ffff9c1a78a78f00 x1874668973148928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.672186:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973148928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.672193:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.672196:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829503.672199:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973148928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829503.672212:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829500/real 1787829503] req@ffff9c1a78a7a580 x1874668973149056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.672225:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973149056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.672235:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.672238:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829503.672242:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973149056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829503.672249:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829500/real 1787829503] req@ffff9c1a78a7b840 x1874668973148288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829503.672261:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973148288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829503.672269:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829503.672273:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829503.672276:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973148288:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829504.183415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829504.183415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829504.439445:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829505.207457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829505.207458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829505.463502:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829505.719514:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829505.719525:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829505.719529:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829505.719539:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5956 00000100:00080000:0.0:1787829505.719551:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829505.719559:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829505.719596:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973149184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829505.719627:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829505.719632:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829505.719634:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829505.719639:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5956 00000100:00080000:0.0:1787829505.719643:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829505.719647:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829505.719657:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829505.719672:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973149312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829505.719684:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829505.719688:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829505.719690:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829505.719694:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5956 00000100:00080000:0.0:1787829505.719698:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829505.719702:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829505.719707:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829505.719719:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973149440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829505.719730:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829505.719733:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829505.719735:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829505.719739:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5956 00000100:00080000:0.0:1787829505.719743:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829505.719747:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829505.719752:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829505.719762:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973149568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829505.719772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829505.719776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829505.719777:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829505.719781:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5956 00000100:00080000:0.0:1787829505.719785:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829505.719789:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829505.719794:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829505.719805:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973149696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829505.719813:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829505.719816:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829505.719818:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829505.719822:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5956 00000100:00080000:0.0:1787829505.719826:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829505.719829:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829505.719834:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829505.719844:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973149824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829505.719852:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829505.719856:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829505.719858:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829505.719861:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5956 00000100:00080000:0.0:1787829505.719864:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829505.719868:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829505.719872:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829505.719881:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973149952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829505.720000:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973149184:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787829505.720065:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829505.720105:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829505/real 1787829505] req@ffff9c1a78860780 x1874668973149184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829505.720125:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973149184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829505.720137:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829505.720142:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829505.720147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973149184:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829505.720155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973149312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829505.720186:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829505.720202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829505/real 1787829505] req@ffff9c1a78862940 x1874668973149312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829505.720216:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973149312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829505.720225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829505.720228:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829505.720232:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973149312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829505.720239:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973149440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829505.720265:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829505.720277:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829505/real 1787829505] req@ffff9c1a78861a40 x1874668973149440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829505.720302:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973149440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829505.720312:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829505.720315:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829505.720319:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973149440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829505.720327:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973149568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829505.720381:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829505.720395:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829505/real 1787829505] req@ffff9c1a78860f00 x1874668973149568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829505.720410:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973149568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829505.720419:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829505.720422:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829505.720425:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973149568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829505.720431:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973149696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829505.720453:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829505.720468:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829505/real 1787829505] req@ffff9c1a788621c0 x1874668973149696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829505.720486:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973149696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829505.720496:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829505.720499:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829505.720502:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973149696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829505.720509:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973149824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829505.720532:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829505.720543:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829505/real 1787829505] req@ffff9c1a78863840 x1874668973149824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829505.720556:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973149824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829505.720565:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829505.720568:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829505.720572:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973149824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829505.720579:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973149952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829505.720600:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829505.720612:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829505/real 1787829505] req@ffff9c1a78862d00 x1874668973149952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829505.720625:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973149952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829505.720639:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829505.720643:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829505.720646:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973149952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829506.231448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829506.231454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829506.487539:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829507.255444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829507.255444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829507.511512:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829508.279443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829508.279467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829508.535468:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829509.303416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829509.303453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829509.559485:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829510.327471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829510.327479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829510.583495:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829510.839444:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829510.839455:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829510.839459:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829510.839469:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5961 00000100:00080000:2.0:1787829510.839481:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829510.839489:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829510.839528:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973150080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829510.839553:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829510.839556:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829510.839557:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829510.839561:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5961 00000100:00080000:2.0:1787829510.839563:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829510.839567:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829510.839575:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829510.839584:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973150208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829510.839591:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829510.839600:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829510.839602:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829510.839605:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5961 00000100:00080000:2.0:1787829510.839608:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829510.839611:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829510.839615:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829510.839622:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973150336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829510.839635:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973150080:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829510.839635:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829510.839644:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829510.839649:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829510.839657:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5961 00000100:00080000:2.0:1787829510.839659:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829510.839662:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829510.839664:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829510.839671:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973150464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829510.839676:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829510.839679:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829510.839681:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829510.839683:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5961 00000100:00080000:2.0:1787829510.839685:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829510.839687:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829510.839690:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829510.839697:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973150592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829510.839705:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829510.839707:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829510.839708:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829510.839712:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5961 00000100:00080000:2.0:1787829510.839715:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829510.839717:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829510.839720:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829510.839726:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973150720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829510.839732:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829510.839734:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829510.839735:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829510.839738:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5961 00000100:00080000:2.0:1787829510.839740:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829510.839745:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829510.839746:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829510.839754:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829510.839761:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973150848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829510.839814:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829510/real 1787829510] req@ffff9c1a49d8cf00 x1874668973150080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829510.839836:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973150080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829510.839845:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829510.839848:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829510.839852:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973150080:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829510.839858:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973150208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829510.839888:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829510.839902:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829510/real 1787829510] req@ffff9c1a49d8c000 x1874668973150208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829510.839914:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973150208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829510.839923:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829510.839927:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829510.839930:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973150208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829510.839961:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973150336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829510.839981:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829510.839991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829510/real 1787829510] req@ffff9c1a49d8c780 x1874668973150336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829510.840003:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973150336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829510.840016:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829510.840019:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829510.840021:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973150336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829510.840025:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973150464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829510.840038:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829510.840045:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829510/real 1787829510] req@ffff9c1a49d8de00 x1874668973150464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829510.840051:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973150464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829510.840057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829510.840059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829510.840060:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973150464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829510.840064:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973150592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829510.840073:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829510.840078:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829510/real 1787829510] req@ffff9c1a49d8da40 x1874668973150592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829510.840084:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973150592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829510.840089:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829510.840090:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829510.840093:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973150592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829510.840097:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973150720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829510.840107:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829510.840112:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829510/real 1787829510] req@ffff9c1a49d8cb40 x1874668973150720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829510.840118:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973150720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829510.840123:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829510.840125:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829510.840126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973150720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829510.840130:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973150848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829510.840138:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829510.840143:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829510/real 1787829510] req@ffff9c1a49d8f0c0 x1874668973150848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829510.840149:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973150848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829510.840153:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829510.840155:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829510.840157:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973150848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829511.351409:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829511.351419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829511.607474:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829512.375474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829512.375474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829512.631504:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829513.399400:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829513.399423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829513.655494:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829514.423480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829514.423480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829514.679549:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829515.447451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829515.447451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829515.703481:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829515.959471:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829515.959482:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829515.959486:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829515.959496:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5966 00000100:00080000:2.0:1787829515.959507:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829515.959515:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829515.959553:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973150976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829515.959586:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829515.959591:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829515.959593:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829515.959598:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5966 00000100:00080000:2.0:1787829515.959603:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829515.959607:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829515.959617:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829515.959631:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973151104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829515.959635:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973150976:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829515.959648:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829515.959651:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829515.959652:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829515.959656:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5966 00000100:00080000:2.0:1787829515.959659:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829515.959661:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829515.959665:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829515.959676:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973151232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829515.959694:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829515.959699:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829515.959701:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829515.959705:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829515.959711:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5966 00000100:00080000:2.0:1787829515.959717:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829515.959722:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829515.959727:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829515.959741:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973151360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829515.959747:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829515/real 1787829515] req@ffff9c1a49d8e1c0 x1874668973150976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829515.959752:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829515.959762:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829515.959765:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829515.959772:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5966 00000100:00080000:2.0:1787829515.959777:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829515.959780:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973150976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829515.959785:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829515.959793:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829515.959798:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829515.959804:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829515.959807:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973151488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829515.959812:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973150976:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829515.959825:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829515.959828:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829515.959830:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829515.959833:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5966 00000100:00080000:2.0:1787829515.959837:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829515.959841:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829515.959845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973151104:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787829515.959846:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829515.959859:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973151616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829515.959868:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829515.959870:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829515.959873:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829515.959877:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5966 00000100:00080000:2.0:1787829515.959881:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829515.959884:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829515.959889:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829515.959896:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829515.959901:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829515/real 1787829515] req@ffff9c1a49d8d680 x1874668973151104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829515.959913:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973151744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829515.959915:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973151104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829515.959949:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829515.959953:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829515.959957:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973151104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829515.959963:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973151232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829515.959979:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829515.959986:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829515/real 1787829515] req@ffff9c1a49d8ed00 x1874668973151232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829515.959998:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973151232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829515.960004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829515.960006:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829515.960008:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973151232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829515.960012:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973151360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829515.960021:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829515.960027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829515/real 1787829515] req@ffff9c1a49d8fc00 x1874668973151360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829515.960035:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973151360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829515.960040:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829515.960042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829515.960043:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973151360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829515.960047:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973151488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829515.960056:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829515.960062:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829515/real 1787829515] req@ffff9c1a49d8e580 x1874668973151488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829515.960069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973151488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829515.960079:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829515.960081:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829515.960083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973151488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829515.960096:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973151616:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829515.960106:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829515.960112:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829515/real 1787829515] req@ffff9c1a49d8c3c0 x1874668973151616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829515.960119:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973151616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829515.960125:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829515.960126:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829515.960128:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973151616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829515.960132:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973151744:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829515.960141:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829515.960146:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829515/real 1787829515] req@ffff9c1a49d8e1c0 x1874668973151744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829515.960153:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973151744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829515.960158:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829515.960160:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829515.960163:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973151744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829516.471448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829516.471448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829516.727520:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829517.495434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829517.495434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829517.751529:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829518.519450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829518.519459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829518.775483:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829519.543438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829519.543504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829519.799536:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829520.567476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829520.567476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829520.823524:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829521.079480:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829521.079491:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829521.079497:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829521.079508:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5971 00000100:00080000:2.0:1787829521.079520:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829521.079527:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829521.079563:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973151872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829521.079594:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829521.079598:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829521.079600:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829521.079604:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5971 00000100:00080000:2.0:1787829521.079608:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829521.079612:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829521.079621:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829521.079634:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973152000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829521.079645:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829521.079649:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829521.079651:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829521.079658:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973151872:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829521.079658:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5971 00000100:00080000:2.0:1787829521.079667:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829521.079672:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829521.079678:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829521.079690:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973152128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829521.079701:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829521.079705:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829521.079707:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829521.079711:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5971 00000100:00080000:2.0:1787829521.079714:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829521.079719:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787829521.079724:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787829521.079727:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829521.079740:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973152256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829521.079751:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829521.079754:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829521.079756:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829521.079759:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5971 00000100:00080000:2.0:1787829521.079763:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829521.079766:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829521.079772:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829521.079780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff9c1a49d8f0c0 x1874668973151872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829521.079786:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973152384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829521.079802:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829521.079806:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829521.079809:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829521.079810:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973151872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829521.079817:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5971 00000100:00080000:2.0:1787829521.079824:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829521.079831:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787829521.079833:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829521.079840:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829521.079845:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973151872:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787829521.079850:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829521.079864:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973152512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829521.079873:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973152000:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829521.079879:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829521.079884:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829521.079886:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829521.079891:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5971 00000800:00000100:1.0:1787829521.079899:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829521.079906:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829521.079914:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff9c1a49d8cb40 x1874668973152000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829521.079915:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829521.079923:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829521.079940:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973152000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829521.079942:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973152640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829521.079966:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829521.079979:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829521.079983:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973152000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829521.079988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973152128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829521.080013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829521.080025:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff9c1a49d8da40 x1874668973152128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829521.080039:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973152128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829521.080048:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829521.080051:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829521.080054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973152128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829521.080058:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973152256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829521.080075:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829521.080083:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff9c1a49d8de00 x1874668973152256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829521.080091:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973152256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829521.080097:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829521.080099:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829521.080101:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973152256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829521.080104:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973152384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829521.080114:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829521.080121:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff9c1a49d8c780 x1874668973152384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829521.080129:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973152384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829521.080134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829521.080135:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829521.080137:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973152384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829521.080150:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973152512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829521.080161:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829521.080167:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff9c1a49d8c000 x1874668973152512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829521.080175:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973152512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829521.080181:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829521.080198:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829521.080200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973152512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829521.080205:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973152640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829521.080219:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829521.080227:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829521/real 1787829521] req@ffff9c1a49d8cf00 x1874668973152640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829521.080238:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973152640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829521.080252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829521.080255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829521.080258:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973152640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829521.591429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829521.591429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829521.847547:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829522.615479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829522.615600:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829522.871549:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829523.639430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829523.639430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829523.895497:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829524.663485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829524.663490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829524.919544:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829525.687442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829525.687454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829525.943449:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829526.199460:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829526.199468:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829526.199471:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829526.199479:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5976 00000100:00080000:0.0:1787829526.199487:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829526.199493:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829526.199524:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973152768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829526.199545:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829526.199548:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829526.199549:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829526.199552:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5976 00000100:00080000:0.0:1787829526.199554:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829526.199557:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829526.199564:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829526.199574:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973152896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829526.199580:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829526.199583:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829526.199584:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829526.199588:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5976 00000100:00080000:0.0:1787829526.199589:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829526.199591:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829526.199594:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829526.199599:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973153024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829526.199607:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829526.199610:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829526.199611:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829526.199613:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5976 00000100:00080000:0.0:1787829526.199615:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829526.199617:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829526.199619:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829526.199625:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973153152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829526.199632:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829526.199635:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829526.199638:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829526.199644:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5976 00000100:00100000:1.0:1787829526.199646:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973152768:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829526.199648:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829526.199654:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829526.199660:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829526.199670:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973153280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829526.199675:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829526.199676:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829526.199678:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829526.199680:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5976 00000100:00080000:0.0:1787829526.199682:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829526.199685:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829526.199689:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829526.199697:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973153408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829526.199703:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829526.199706:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829526.199708:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829526.199714:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5976 00000100:00080000:0.0:1787829526.199729:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829526.199734:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829526.199742:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829526.199748:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829526.199761:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973153536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829526.199804:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff9c1a78860b40 x1874668973152768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829526.199826:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973152768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829526.199838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829526.199843:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829526.199847:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973152768:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829526.199856:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973152896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829526.199881:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829526.199896:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff9c1a78862d00 x1874668973152896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829526.199909:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973152896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829526.199917:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829526.199921:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829526.199924:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973152896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829526.199931:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973153024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829526.199951:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829526.199962:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff9c1a78863840 x1874668973153024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829526.199974:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973153024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829526.199982:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829526.199986:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829526.199989:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973153024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829526.200021:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973153152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829526.200042:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829526.200053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff9c1a788621c0 x1874668973153152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829526.200074:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973153152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829526.200087:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829526.200090:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829526.200094:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973153152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829526.200100:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973153280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829526.200116:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829526.200125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff9c1a78860f00 x1874668973153280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829526.200142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973153280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829526.200162:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829526.200165:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829526.200169:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973153280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829526.200175:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973153408:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829526.200192:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829526.200202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff9c1a78861a40 x1874668973153408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829526.200213:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973153408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829526.200220:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829526.200222:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829526.200225:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973153408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829526.200229:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973153536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829526.200242:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829526.200248:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829526/real 1787829526] req@ffff9c1a78862940 x1874668973153536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829526.200255:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973153536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829526.200260:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829526.200262:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829526.200264:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973153536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829526.711427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829526.711427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829526.967512:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829527.735468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829527.735468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829527.991470:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829528.759434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829528.759435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829529.015561:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829529.783494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829529.783494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829530.039491:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829530.807429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829530.807431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829531.063502:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829531.319446:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829531.319454:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829531.319459:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829531.319471:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5981 00000100:00080000:0.0:1787829531.319481:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829531.319486:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829531.319518:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973153664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829531.319545:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829531.319548:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829531.319550:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829531.319554:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5981 00000100:00080000:0.0:1787829531.319557:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829531.319560:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829531.319572:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829531.319574:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973153664:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829531.319586:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973153792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829531.319595:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829531.319598:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829531.319600:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829531.319602:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5981 00000100:00080000:0.0:1787829531.319606:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829531.319609:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829531.319613:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829531.319628:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973153920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829531.319633:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829531.319651:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829531.319655:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829531.319658:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829531.319664:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5981 00000100:00080000:0.0:1787829531.319670:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829531.319672:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff9c1a78860780 x1874668973153664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829531.319678:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829531.319688:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829531.319695:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973153664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829531.319703:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973154048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829531.319706:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829531.319715:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829531.319716:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829531.319721:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829531.319722:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973153664:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829531.319724:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829531.319734:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5981 00000100:00080000:0.0:1787829531.319738:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829531.319741:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829531.319744:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829531.319750:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973153792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829531.319755:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973154176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829531.319769:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829531.319774:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829531.319775:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829531.319778:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829531.319782:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5981 00000100:00080000:0.0:1787829531.319787:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829531.319790:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff9c1a78861e00 x1874668973153792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829531.319792:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829531.319799:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829531.319811:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973153792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829531.319813:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973154304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829531.319828:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829531.319830:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829531.319831:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829531.319835:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5981 00000100:00080000:0.0:1787829531.319839:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829531.319845:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829531.319850:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829531.319853:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829531.319858:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973154432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829531.319861:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829531.319869:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973153792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829531.319878:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973153920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829531.319902:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829531.319911:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff9c1a788612c0 x1874668973153920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829531.319920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973153920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829531.319926:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829531.319928:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829531.319930:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973153920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829531.319933:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973154048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829531.319944:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829531.319949:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff9c1a78863c00 x1874668973154048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829531.319957:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973154048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829531.319962:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829531.319964:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829531.319967:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973154048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829531.319979:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973154176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829531.319990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829531.319997:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff9c1a788630c0 x1874668973154176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829531.320005:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973154176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829531.320010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829531.320012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829531.320015:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973154176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829531.320019:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973154304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829531.320029:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829531.320037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff9c1a78862580 x1874668973154304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829531.320045:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973154304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829531.320051:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829531.320054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829531.320058:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973154304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829531.320064:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973154432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829531.320078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829531.320091:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829531/real 1787829531] req@ffff9c1a78861680 x1874668973154432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829531.320100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973154432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829531.320106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829531.320109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829531.320126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973154432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829531.831466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829531.831467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829532.087531:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829532.855452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829532.855457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829533.111493:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829533.879454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829533.879454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829534.135507:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829534.903452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829534.903452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829535.159509:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829535.927456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829535.927456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829536.183546:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829536.439446:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.439454:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829536.439457:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.439466:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5986 00000100:00080000:0.0:1787829536.439475:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829536.439481:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829536.439511:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973154560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829536.439534:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.439537:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829536.439538:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.439542:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5986 00000100:00080000:0.0:1787829536.439544:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829536.439546:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829536.439553:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829536.439563:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973154688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829536.439572:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.439575:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829536.439576:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.439580:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5986 00000100:00080000:0.0:1787829536.439585:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829536.439585:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973154560:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829536.439588:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829536.439594:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829536.439601:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973154816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829536.439608:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.439611:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829536.439613:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.439615:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5986 00000100:00080000:0.0:1787829536.439618:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829536.439620:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829536.439623:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829536.439629:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973154944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829536.439635:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.439637:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829536.439639:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.439641:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5986 00000100:00080000:0.0:1787829536.439644:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829536.439646:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829536.439649:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829536.439656:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973155072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829536.439662:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829536.439670:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829536.439673:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829536.439675:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.439680:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5986 00000100:00080000:0.0:1787829536.439685:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829536.439690:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829536.439696:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829536.439706:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff9c1a78860000 x1874668973154560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829536.439712:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973155200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829536.439738:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829536.439740:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973154560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829536.439743:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829536.439746:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829536.439749:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5986 00000100:00080000:0.0:1787829536.439752:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829536.439756:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829536.439765:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829536.439769:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829536.439775:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829536.439778:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973155328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829536.439782:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973154560:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829536.439810:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973154688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829536.439837:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829536.439852:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff9c1a78861680 x1874668973154688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829536.439867:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973154688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829536.439876:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829536.439878:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829536.439880:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973154688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829536.439885:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973154816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829536.439901:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829536.439909:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff9c1a78862580 x1874668973154816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829536.439917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973154816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829536.439925:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829536.439929:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829536.439932:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973154816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829536.439938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973154944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829536.439959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829536.439972:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff9c1a788630c0 x1874668973154944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829536.439981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973154944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829536.439987:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829536.439989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829536.439992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973154944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829536.439996:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973155072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829536.440010:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829536.440016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff9c1a78863c00 x1874668973155072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829536.440024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973155072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829536.440029:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829536.440030:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829536.440032:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973155072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829536.440036:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973155200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829536.440046:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829536.440052:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff9c1a788612c0 x1874668973155200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829536.440059:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973155200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829536.440064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829536.440066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829536.440067:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973155200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829536.440071:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973155328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829536.440097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829536.440106:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829536/real 1787829536] req@ffff9c1a78861e00 x1874668973155328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829536.440114:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973155328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829536.440119:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829536.440121:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829536.440123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973155328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829536.951449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829536.951544:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829537.207491:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829537.975413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829537.975425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829538.231536:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829538.999444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829538.999444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829539.255512:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829540.023456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829540.023458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829540.279513:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829541.047424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829541.047456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829541.303525:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829541.559433:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829541.559447:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829541.559452:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829541.559464:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5991 00000100:00080000:0.0:1787829541.559478:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829541.559485:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829541.559533:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973155456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829541.559568:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829541.559571:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829541.559573:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829541.559577:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5991 00000100:00080000:0.0:1787829541.559581:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829541.559585:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829541.559595:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829541.559608:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973155456:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829541.559609:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973155584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829541.559631:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829541.559633:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829541.559635:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829541.559641:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5991 00000100:00080000:0.0:1787829541.559645:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829541.559650:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829541.559656:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829541.559680:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973155712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829541.559680:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829541.559700:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829541.559703:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829541.559704:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829541.559709:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5991 00000100:00080000:0.0:1787829541.559714:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829541.559723:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829541.559724:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829541/real 1787829541] req@ffff9c1a78860780 x1874668973155456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829541.559737:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829541.559747:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973155456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829541.559760:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973155840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829541.559761:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829541.559774:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829541.559787:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973155456:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829541.559808:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829541.559811:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829541.559813:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829541.559817:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5991 00000100:00080000:0.0:1787829541.559820:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829541.559824:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973155584:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829541.559832:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829541.559841:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829541.559852:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829541.559864:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973155968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829541.559873:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829541/real 1787829541] req@ffff9c1a78862940 x1874668973155584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829541.559885:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829541.559890:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829541.559893:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829541.559898:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973155584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829541.559904:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5991 00000100:00080000:0.0:1787829541.559912:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829541.559921:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829541.559928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829541.559930:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829541.559945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829541.559950:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973156096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829541.559954:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973155584:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829541.559972:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829541.559977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829541.559978:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973155712:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829541.559982:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829541.559989:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5991 00000100:00080000:0.0:1787829541.559992:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829541.559995:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829541.559998:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829541.560011:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829541.560012:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973156224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829541.560026:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829541/real 1787829541] req@ffff9c1a78861a40 x1874668973155712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829541.560039:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973155712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829541.560066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829541.560069:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829541.560072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973155712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829541.560078:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973155840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829541.560095:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829541.560103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829541/real 1787829541] req@ffff9c1a78860f00 x1874668973155840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829541.560110:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973155840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829541.560116:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829541.560119:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829541.560121:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973155840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829541.560134:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973155968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829541.560145:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829541.560151:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829541/real 1787829541] req@ffff9c1a788621c0 x1874668973155968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829541.560159:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973155968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829541.560164:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829541.560166:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829541.560168:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973155968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829541.560172:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973156096:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829541.560182:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829541.560193:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829541/real 1787829541] req@ffff9c1a78863840 x1874668973156096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829541.560205:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973156096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829541.560213:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829541.560217:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829541.560220:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973156096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829541.560226:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973156224:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829541.560246:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829541.560258:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829541/real 1787829541] req@ffff9c1a78862d00 x1874668973156224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829541.560271:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973156224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829541.560279:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829541.560283:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829541.560286:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973156224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829542.071478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829542.071478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829542.327567:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829543.095447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829543.095447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829543.351487:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829544.119486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829544.119492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829544.375503:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829545.143463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829545.143463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829545.399509:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829546.167407:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829546.167407:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829546.423566:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829546.679504:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829546.679512:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829546.679516:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829546.679525:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 5997 00000100:00080000:3.0:1787829546.679535:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829546.679541:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829546.679576:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973156352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829546.679599:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829546.679602:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829546.679603:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829546.679606:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5997 00000100:00080000:3.0:1787829546.679608:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829546.679611:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829546.679619:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829546.679630:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973156480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829546.679638:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829546.679640:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973156352:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829546.679642:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829546.679646:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829546.679653:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 5997 00000100:00080000:3.0:1787829546.679655:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829546.679657:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829546.679660:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829546.679668:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973156608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829546.679676:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829546.679678:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829546.679679:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829546.679682:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5997 00000100:00080000:3.0:1787829546.679684:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829546.679686:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829546.679689:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829546.679696:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973156736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829546.679701:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829546.679704:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829546.679705:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829546.679708:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5997 00000100:00080000:3.0:1787829546.679718:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829546.679721:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829546.679723:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829546.679730:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973156864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829546.679736:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829546.679738:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829546.679739:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829546.679741:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5997 00000100:00080000:3.0:1787829546.679743:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829546.679745:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829546.679747:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829546.679752:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973156992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829546.679758:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829546.679760:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829546.679761:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829546.679763:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 5997 00000100:00080000:3.0:1787829546.679764:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829546.679766:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829546.679768:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829546.679774:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973157120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829546.679782:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829546.679827:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829546/real 1787829546] req@ffff9c1a7895c3c0 x1874668973156352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829546.679847:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973156352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829546.679859:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829546.679863:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829546.679867:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973156352:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829546.679889:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973156480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829546.679913:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829546.679926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829546/real 1787829546] req@ffff9c1a7895c780 x1874668973156480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829546.679940:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973156480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829546.679955:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829546.679959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829546.679963:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973156480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829546.679970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973156608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829546.679989:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829546.680001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829546/real 1787829546] req@ffff9c1a7895e940 x1874668973156608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829546.680033:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973156608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829546.680042:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829546.680045:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829546.680049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973156608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829546.680055:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973156736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829546.680078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829546.680091:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829546/real 1787829546] req@ffff9c1a7895cf00 x1874668973156736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829546.680103:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973156736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829546.680113:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829546.680116:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829546.680120:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973156736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829546.680127:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973156864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829546.680147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829546.680159:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829546/real 1787829546] req@ffff9c1a7895e580 x1874668973156864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829546.680172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973156864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829546.680181:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829546.680184:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829546.680188:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973156864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829546.680194:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973156992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829546.680214:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829546.680225:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829546/real 1787829546] req@ffff9c1a7895c000 x1874668973156992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829546.680238:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973156992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829546.680247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829546.680250:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829546.680254:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973156992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829546.680260:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973157120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829546.680281:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829546.680293:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829546/real 1787829546] req@ffff9c1a7895f480 x1874668973157120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829546.680304:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973157120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829546.680314:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829546.680317:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829546.680320:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973157120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829547.191447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829547.191447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829547.447452:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829548.215414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829548.215431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829548.471523:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829549.239455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829549.239455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829549.495513:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829550.263423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829550.263435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829550.519501:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829551.287467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829551.287467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829551.543536:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829551.799428:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829551.799436:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829551.799439:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829551.799447:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6002 00000100:00080000:3.0:1787829551.799458:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829551.799463:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829551.799496:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973157248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829551.799526:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829551.799531:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829551.799533:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829551.799538:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6002 00000100:00080000:3.0:1787829551.799543:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829551.799547:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829551.799557:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829551.799570:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973157248:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829551.799571:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973157376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829551.799586:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829551.799588:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829551.799590:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829551.799593:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6002 00000100:00080000:3.0:1787829551.799596:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829551.799599:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829551.799603:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829551.799611:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973157504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829551.799619:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829551.799622:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829551.799623:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829551.799625:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6002 00000100:00080000:3.0:1787829551.799628:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829551.799632:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829551.799635:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829551.799640:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829551.799653:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973157632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829551.799664:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829551.799669:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829551.799670:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829551/real 1787829551] req@ffff9c1a7895d2c0 x1874668973157248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829551.799682:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829551.799687:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973157248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829551.799691:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6002 00000100:00080000:2.0:1787829551.799702:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829551.799702:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829551.799709:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829551.799711:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829551.799716:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973157248:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787829551.799719:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829551.799732:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973157760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829551.799739:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973157376:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829551.799743:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829551.799747:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829551.799749:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829551.799754:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6002 00000100:00080000:3.0:1787829551.799758:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829551.799762:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829551.799763:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829551.799769:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829551.799776:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829551/real 1787829551] req@ffff9c1a7895cb40 x1874668973157376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829551.799781:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973157888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829551.799813:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829551.799815:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829551.799817:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829551.799821:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973157376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829551.799822:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6002 00000100:00080000:3.0:1787829551.799826:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829551.799831:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829551.799836:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829551.799846:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829551.799847:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973158016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829551.799856:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829551.799862:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973157376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829551.799868:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973157504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829551.799886:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829551.799894:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829551/real 1787829551] req@ffff9c1a7895da40 x1874668973157504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829551.799903:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973157504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829551.799908:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829551.799910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829551.799912:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973157504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829551.799916:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973157632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829551.799926:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829551.799932:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829551/real 1787829551] req@ffff9c1a7895e1c0 x1874668973157632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829551.799939:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973157632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829551.799944:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829551.799946:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829551.799948:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973157632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829551.799960:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973157760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829551.799987:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829551.800000:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829551/real 1787829551] req@ffff9c1a7895fc00 x1874668973157760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829551.800013:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973157760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829551.800021:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829551.800024:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829551.800027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973157760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829551.800034:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973157888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829551.800055:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829551.800067:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829551/real 1787829551] req@ffff9c1a7895f840 x1874668973157888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829551.800081:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973157888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829551.800089:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829551.800092:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829551.800096:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973157888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829551.800102:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973158016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829551.800121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829551.800133:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829551/real 1787829551] req@ffff9c1a7895f0c0 x1874668973158016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829551.800145:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973158016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829551.800153:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829551.800156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829551.800160:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973158016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829552.311415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829552.311439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829552.567493:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829553.335500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829553.335500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829553.591550:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829554.359466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829554.359474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829554.615459:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829555.383451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829555.383451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829555.639509:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829556.407441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829556.407443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829556.663494:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829556.919488:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829556.919499:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829556.919504:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829556.919514:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6007 00000100:00080000:2.0:1787829556.919525:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829556.919533:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829556.919573:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973158144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829556.919609:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829556.919613:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829556.919615:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829556.919620:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6007 00000100:00080000:2.0:1787829556.919624:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829556.919629:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829556.919639:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829556.919653:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973158144:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829556.919657:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973158272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829556.919672:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829556.919675:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829556.919677:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829556.919682:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6007 00000100:00080000:2.0:1787829556.919687:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829556.919692:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829556.919697:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829556.919711:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973158400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829556.919718:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829556.919731:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829556.919735:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829556.919738:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829556.919743:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6007 00000100:00080000:2.0:1787829556.919747:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829556.919752:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829556.919759:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829556.919766:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829556/real 1787829556] req@ffff9c1a49d8cf00 x1874668973158144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829556.919774:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973158528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829556.919796:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829556.919802:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973158144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829556.919802:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829556.919807:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829556.919818:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6007 00000100:00080000:1.0:1787829556.919823:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829556.919825:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829556.919833:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829556.919834:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829556.919843:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973158144:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787829556.919843:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829556.919864:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973158656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829556.919875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973158272:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829556.919879:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829556.919885:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829556.919887:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829556.919893:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6007 00000100:00080000:2.0:1787829556.919899:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829556.919908:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829556.919912:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829556.919919:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829556.919927:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829556/real 1787829556] req@ffff9c1a49d8c000 x1874668973158272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829556.919949:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973158784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829556.919964:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973158272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829556.919965:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829556.919977:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829556.919981:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829556.919985:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829556.919990:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6007 00000100:00080000:1.0:1787829556.919994:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829556.919998:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829556.920001:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973158272:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829556.920008:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829556.920015:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973158400:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787829556.920025:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829556.920044:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973158912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829556.920045:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829556.920061:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829556/real 1787829556] req@ffff9c1a49d8c780 x1874668973158400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829556.920077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973158400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829556.920093:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829556.920098:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829556.920101:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973158400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829556.920108:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973158528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829556.920129:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829556.920142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829556/real 1787829556] req@ffff9c1a49d8de00 x1874668973158528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829556.920154:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973158528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829556.920163:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829556.920166:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829556.920170:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973158528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829556.920189:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973158656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829556.920209:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829556.920221:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829556/real 1787829556] req@ffff9c1a49d8da40 x1874668973158656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829556.920233:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973158656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829556.920242:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829556.920246:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829556.920250:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973158656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829556.920256:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973158784:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829556.920277:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829556.920289:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829556/real 1787829556] req@ffff9c1a49d8cb40 x1874668973158784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829556.920301:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973158784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829556.920310:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829556.920313:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829556.920317:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973158784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829556.920323:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973158912:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829556.920343:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829556.920397:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829556/real 1787829556] req@ffff9c1a49d8f0c0 x1874668973158912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829556.920411:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973158912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829556.920420:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829556.920424:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829556.920427:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973158912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829557.431475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829557.431475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829557.687567:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829558.455484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829558.455484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829558.711531:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829559.479463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829559.479463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829559.735590:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829560.503428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829560.503428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829560.759513:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829561.527434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829561.527461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:2.0:1787829561.783502:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829562.039509:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829562.039520:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829562.039525:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829562.039536:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6012 00000100:00080000:1.0:1787829562.039548:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829562.039555:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829562.039608:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973159040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829562.039637:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829562.039640:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829562.039642:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829562.039645:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6012 00000100:00080000:1.0:1787829562.039648:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829562.039651:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829562.039661:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829562.039673:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973159168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829562.039684:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829562.039691:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829562.039694:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829562.039703:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6012 00000100:00100000:2.0:1787829562.039703:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973159040:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829562.039712:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829562.039723:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829562.039727:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829562.039735:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973159296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829562.039746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829562.039749:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829562.039752:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829562.039756:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6012 00000100:00080000:1.0:1787829562.039759:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829562.039763:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829562.039767:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829562.039775:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973159424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829562.039782:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829562.039785:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829562.039789:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829562.039790:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829562.039800:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6012 00000100:00080000:1.0:1787829562.039805:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829562.039809:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829562.039815:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829562.039833:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973159552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829562.039840:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff9c1a78a7b840 x1874668973159040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829562.039862:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829562.039868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829562.039869:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973159040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829562.039874:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829562.039891:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6012 00000100:00080000:2.0:1787829562.039892:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829562.039915:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829562.039917:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829562.039925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973159040:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829562.039932:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829562.039943:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829562.039953:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973159168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829562.039970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973159680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829562.039983:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829562.039986:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829562.039992:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829562.039996:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829562.040000:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6012 00000100:00080000:1.0:1787829562.040005:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829562.040009:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829562.040014:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829562.040015:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff9c1a78a7a580 x1874668973159168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829562.040031:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973159168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829562.040033:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973159808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829562.040082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829562.040088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829562.040097:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973159168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829562.040105:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973159296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829562.040127:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829562.040134:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff9c1a78a78f00 x1874668973159296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829562.040142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973159296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829562.040148:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829562.040150:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829562.040152:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973159296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829562.040156:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973159424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829562.040166:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829562.040172:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff9c1a78a7b480 x1874668973159424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829562.040179:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973159424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829562.040185:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829562.040186:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829562.040188:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973159424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829562.040192:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973159552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829562.040202:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829562.040208:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff9c1a78a79a40 x1874668973159552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829562.040220:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973159552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829562.040225:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829562.040227:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829562.040229:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973159552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829562.040242:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973159680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829562.040253:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829562.040259:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff9c1a78a78000 x1874668973159680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829562.040266:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973159680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829562.040271:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829562.040274:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829562.040275:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973159680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829562.040279:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973159808:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829562.040288:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829562.040302:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829562/real 1787829562] req@ffff9c1a78a792c0 x1874668973159808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829562.040309:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973159808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829562.040318:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829562.040320:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829562.040322:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973159808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829562.551428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829562.551429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829562.807490:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829563.575427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829563.575428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829563.831606:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829564.599432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829564.599432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829564.855535:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829565.623461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829565.623511:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829565.880756:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829566.647450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829566.647450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829566.903486:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829567.159476:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829567.159489:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829567.159493:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829567.159505:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6017 00000100:00080000:2.0:1787829567.159517:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829567.159524:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829567.159566:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973159936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829567.159596:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829567.159600:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829567.159602:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829567.159606:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6017 00000100:00080000:2.0:1787829567.159609:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829567.159612:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829567.159623:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829567.159635:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973160064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829567.159646:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829567.159653:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829567.159656:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973159936:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829567.159657:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829567.159672:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6017 00000100:00080000:2.0:1787829567.159676:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829567.159679:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829567.159683:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829567.159698:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973160192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829567.159706:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829567.159709:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829567.159711:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829567.159715:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6017 00000100:00080000:2.0:1787829567.159717:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829567.159720:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829567.159723:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829567.159729:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973160320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829567.159735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829567.159738:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829567.159739:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829567.159742:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6017 00000100:00080000:2.0:1787829567.159745:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829567.159749:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829567.159752:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829567.159760:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973160448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829567.159767:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829567.159770:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829567.159771:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829567.159776:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6017 00000100:00080000:2.0:1787829567.159780:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829567.159784:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829567.159788:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829567.159798:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973160576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829567.159804:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829567.159808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829567.159811:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829567.159814:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6017 00000100:00080000:2.0:1787829567.159817:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829567.159822:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829567.159828:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829567.159843:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973160704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829567.159853:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973160064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829567.159911:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973160192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829567.159939:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973160320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829567.159965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973160448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829567.159977:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973160576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829567.159995:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973160704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829567.671449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829567.671449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829567.927545:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829568.695458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829568.695458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829568.951432:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829569.719413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829569.719442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829569.975515:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829570.231592:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:3.0:1787829570.231638:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829570.231660:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829570.231705:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829570.231714:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829570.231722:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829570.231729:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829570.231737:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829570.231745:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787829570.231833:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829570] req@ffff9c1a49d8f0c0 x1874668973160704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829570.231871:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973160704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829570.231883:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829570.231887:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829570.231892:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973160704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829570.231903:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829570] req@ffff9c1a49d8c3c0 x1874668973160064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829570.231924:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973160064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829570.231932:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829570.231936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829570.231940:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973160064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829570.231945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829570] req@ffff9c1a49d8e580 x1874668973160192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829570.231957:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973160192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829570.231965:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829570.231968:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829570.231971:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973160192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829570.231977:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829570] req@ffff9c1a49d8fc00 x1874668973160320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829570.231988:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973160320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829570.231997:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829570.232001:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829570.232004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973160320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829570.232011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829570] req@ffff9c1a49d8ed00 x1874668973160448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829570.232023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973160448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829570.232031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829570.232035:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829570.232039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973160448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829570.232046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829570] req@ffff9c1a49d8d680 x1874668973160576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829570.232058:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973160576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829570.232066:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829570.232070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829570.232073:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973160576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829570.232079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829567/real 1787829570] req@ffff9c1a49d8e1c0 x1874668973159936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829570.232091:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973159936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829570.232098:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829570.232101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829570.232104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973159936:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829570.743481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829570.743852:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829570.999501:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829571.767476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829571.767537:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829572.023512:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829572.279443:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829572.279451:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829572.279454:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829572.279463:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6022 00000100:00080000:2.0:1787829572.279471:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829572.279477:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829572.279512:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973160832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829572.279535:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829572.279539:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829572.279540:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829572.279543:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6022 00000100:00080000:2.0:1787829572.279545:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829572.279548:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829572.279555:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829572.279563:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973160960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829572.279576:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829572.279578:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829572.279580:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829572.279583:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6022 00000100:00080000:2.0:1787829572.279585:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829572.279588:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829572.279591:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829572.279600:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973161088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829572.279602:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973160832:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829572.279622:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829572.279624:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829572.279625:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829572.279629:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6022 00000100:00080000:2.0:1787829572.279633:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829572.279637:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829572.279642:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829572.279656:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973161216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829572.279666:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829572.279670:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829572.279674:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829572.279675:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829572.279683:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6022 00000100:00080000:2.0:1787829572.279687:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829572.279690:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829572.279694:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829572.279703:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973161344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829572.279710:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829572.279713:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829572.279714:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829572.279717:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6022 00000100:00080000:2.0:1787829572.279720:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829572.279724:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829572.279730:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829572] req@ffff9c1a49d8cb40 x1874668973160832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829572.279731:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829572.279750:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973161472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829572.279753:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973160832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829572.279770:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829572.279775:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829572.279776:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829572.279778:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829572.279785:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829572.279786:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6022 00000100:00080000:2.0:1787829572.279794:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829572.279796:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973160832:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829572.279801:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829572.279809:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829572.279810:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973160960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829572.279842:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973161600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829572.279859:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829572.279877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829572] req@ffff9c1a49d8da40 x1874668973160960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829572.279890:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973160960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829572.279898:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829572.279907:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829572.279910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973160960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829572.279938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973161088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829572.279960:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829572.279970:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829572] req@ffff9c1a49d8de00 x1874668973161088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829572.279981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973161088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829572.279989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829572.279992:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829572.279995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973161088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829572.280002:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973161216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829572.280022:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829572.280040:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829572] req@ffff9c1a49d8c780 x1874668973161216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829572.280053:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973161216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829572.280062:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829572.280066:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829572.280069:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973161216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829572.280076:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973161344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829572.280098:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829572.280110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829572] req@ffff9c1a49d8c000 x1874668973161344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829572.280131:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973161344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829572.280139:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829572.280142:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829572.280146:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973161344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829572.280152:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973161472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829572.280174:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829572.280187:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829572] req@ffff9c1a49d8cf00 x1874668973161472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829572.280200:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973161472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829572.280209:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829572.280213:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829572.280217:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973161472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829572.280223:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973161600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829572.280241:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829572.280251:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829572/real 1787829572] req@ffff9c1a49d8e1c0 x1874668973161600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829572.280263:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973161600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829572.280276:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829572.280280:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829572.280283:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973161600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829572.791496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829572.791522:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829573.047542:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829573.815490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829573.815502:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829574.071518:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829574.839481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829574.839506:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829575.095573:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829575.863489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829575.863537:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829576.119481:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829576.887455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829576.887455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829577.143522:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829577.399459:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829577.399467:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829577.399471:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829577.399479:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6027 00000100:00080000:2.0:1787829577.399489:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829577.399494:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829577.399531:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973161728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829577.399566:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829577.399569:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829577.399570:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829577.399574:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6027 00000100:00080000:2.0:1787829577.399577:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829577.399581:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829577.399589:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829577.399600:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973161856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829577.399609:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829577.399611:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973161728:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829577.399613:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829577.399616:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829577.399619:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6027 00000100:00080000:2.0:1787829577.399621:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829577.399627:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829577.399630:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829577.399641:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973161984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829577.399652:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829577.399656:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829577.399658:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829577.399662:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6027 00000100:00080000:2.0:1787829577.399667:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829577.399673:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829577.399678:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829577.399683:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829577.399694:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973162112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829577.399702:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829577.399705:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829577.399706:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829577.399709:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6027 00000100:00080000:2.0:1787829577.399711:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829577.399714:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829577.399717:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829577.399726:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973162240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829577.399731:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff9c1a49d8d680 x1874668973161728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829577.399736:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829577.399743:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829577.399746:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829577.399752:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6027 00000100:00080000:2.0:1787829577.399755:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829577.399758:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829577.399762:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973161728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829577.399762:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829577.399774:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973162368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829577.399777:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.399786:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829577.399806:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973161728:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829577.399815:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829577.399818:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829577.399820:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829577.399824:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6027 00000100:00080000:2.0:1787829577.399827:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829577.399830:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829577.399834:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829577.399835:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973161856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829577.399846:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973162496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829577.399864:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829577.399877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff9c1a49d8ed00 x1874668973161856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.399888:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973161856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.399894:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.399897:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829577.399899:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973161856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829577.399903:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973161984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829577.399915:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829577.399922:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff9c1a49d8fc00 x1874668973161984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.399929:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973161984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.399935:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.399937:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829577.399939:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973161984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829577.399942:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973162112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829577.399953:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829577.399958:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff9c1a49d8e580 x1874668973162112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.399966:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973162112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.399976:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.399978:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829577.399980:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973162112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829577.399983:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973162240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829577.400002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829577.400009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff9c1a49d8c3c0 x1874668973162240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.400016:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973162240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.400022:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.400024:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829577.400026:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973162240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829577.400030:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973162368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829577.400040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829577.400053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff9c1a49d8f0c0 x1874668973162368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.400060:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973162368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.400065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.400067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829577.400070:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973162368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829577.400084:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973162496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829577.400096:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829577.400102:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829577/real 1787829577] req@ffff9c1a49d8d680 x1874668973162496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829577.400110:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973162496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829577.400115:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829577.400117:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829577.400118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973162496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829577.911426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829577.911433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829578.167502:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829578.935456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829578.935456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829579.191598:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829579.959419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829579.959429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829580.215525:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829580.983442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829580.983451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829581.239489:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829582.007419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829582.007425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829582.263488:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829582.519457:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829582.519468:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829582.519478:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829582.519489:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6032 00000100:00080000:0.0:1787829582.519500:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829582.519508:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829582.519543:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973162624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829582.519568:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829582.519572:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829582.519574:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829582.519579:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6032 00000100:00080000:0.0:1787829582.519583:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829582.519587:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829582.519597:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829582.519613:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973162752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829582.519623:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829582.519625:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829582.519627:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829582.519630:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6032 00000100:00080000:0.0:1787829582.519633:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829582.519636:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829582.519640:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829582.519647:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973162624:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829582.519651:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973162880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829582.519661:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829582.519663:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829582.519664:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829582.519667:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6032 00000100:00080000:0.0:1787829582.519669:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829582.519672:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829582.519675:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829582.519689:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973163008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829582.519695:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829582.519697:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829582.519698:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829582.519701:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6032 00000100:00080000:0.0:1787829582.519704:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829582.519708:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829582.519709:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829582.519716:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829582.519725:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973163136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829582.519735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829582.519738:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829582.519741:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829582.519745:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6032 00000100:00100000:1.0:1787829582.519749:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829582/real 1787829582] req@ffff9c1a78860b40 x1874668973162624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829582.519752:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829582.519775:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829582.519784:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829582.519801:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973163264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829582.519807:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973162624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829582.519812:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829582.519820:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829582.519823:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829582.519830:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6032 00000100:00080000:1.0:1787829582.519832:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829582.519834:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829582.519838:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829582.519842:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829582.519848:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973162624:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829582.519850:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829582.519857:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973162752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829582.519862:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973163392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829582.519884:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829582.519898:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829582/real 1787829582] req@ffff9c1a78862d00 x1874668973162752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829582.519906:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973162752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829582.519912:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829582.519914:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829582.519916:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973162752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829582.519934:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973162880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829582.519945:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829582.519951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829582/real 1787829582] req@ffff9c1a78863840 x1874668973162880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829582.519958:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973162880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829582.519968:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829582.519970:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829582.519972:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973162880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829582.519975:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973163008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829582.519984:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829582.519990:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829582/real 1787829582] req@ffff9c1a788621c0 x1874668973163008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829582.519996:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973163008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829582.520002:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829582.520004:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829582.520005:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973163008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829582.520009:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973163136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829582.520018:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829582.520024:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829582/real 1787829582] req@ffff9c1a78860f00 x1874668973163136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829582.520031:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973163136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829582.520036:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829582.520037:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829582.520040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973163136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829582.520044:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973163264:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829582.520054:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829582.520059:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829582/real 1787829582] req@ffff9c1a78861a40 x1874668973163264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829582.520066:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973163264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829582.520071:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829582.520073:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829582.520075:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973163264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829582.520078:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973163392:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829582.520087:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829582.520092:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829582/real 1787829582] req@ffff9c1a78862940 x1874668973163392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829582.520100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973163392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829582.520104:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829582.520106:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829582.520108:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973163392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829583.031457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829583.031457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829583.287576:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829584.055436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829584.055436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829584.311495:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829585.079461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829585.079473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829585.335536:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829586.103459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829586.103459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829586.359464:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829587.127466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829587.127476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829587.383795:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829587.639465:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829587.639473:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829587.639477:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829587.639485:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6037 00000100:00080000:0.0:1787829587.639495:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829587.639501:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829587.639532:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973163520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829587.639562:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829587.639566:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829587.639568:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829587.639572:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6037 00000100:00080000:0.0:1787829587.639577:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829587.639581:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829587.639590:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829587.639604:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973163648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829587.639617:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829587.639622:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829587.639622:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973163520:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829587.639626:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829587.639636:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6037 00000100:00080000:0.0:1787829587.639640:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829587.639644:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829587.639649:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829587.639663:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973163776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829587.639674:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829587.639678:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829587.639680:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829587.639684:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6037 00000100:00080000:0.0:1787829587.639688:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829587.639695:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787829587.639695:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787829587.639703:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829587.639715:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973163904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829587.639742:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829587.639745:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829587.639748:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829587.639750:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829587/real 1787829587] req@ffff9c1a78860780 x1874668973163520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829587.639754:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6037 00000100:00080000:0.0:1787829587.639761:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829587.639768:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829587.639776:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829587.639779:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973163520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829587.639790:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829587.639792:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973164032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829587.639795:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829587.639802:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973163520:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829587.639813:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829587.639817:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829587.639820:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829587.639825:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6037 00000100:00100000:1.0:1787829587.639828:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973163648:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829587.639830:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829587.639838:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829587.639844:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829587.639854:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829587.639859:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973164160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829587.639867:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829587/real 1787829587] req@ffff9c1a78861e00 x1874668973163648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829587.639871:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829587.639880:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829587.639884:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829587.639888:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973163648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829587.639891:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6037 00000100:00080000:0.0:1787829587.639900:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829587.639902:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829587.639910:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787829587.639910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829587.639916:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973163648:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787829587.639920:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829587.639923:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973163776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829587.639938:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973164288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829587.639943:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829587.639953:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829587/real 1787829587] req@ffff9c1a788612c0 x1874668973163776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829587.639962:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973163776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829587.639969:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829587.639972:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829587.639974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973163776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829587.639979:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973163904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829587.639995:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829587.640002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829587/real 1787829587] req@ffff9c1a78863c00 x1874668973163904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829587.640009:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973163904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829587.640015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829587.640016:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829587.640018:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973163904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829587.640022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973164032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829587.640032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829587.640038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829587/real 1787829587] req@ffff9c1a788630c0 x1874668973164032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829587.640046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973164032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829587.640051:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829587.640053:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829587.640054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973164032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829587.640065:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973164160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829587.640076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829587.640082:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829587/real 1787829587] req@ffff9c1a78862580 x1874668973164160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829587.640090:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973164160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829587.640095:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829587.640096:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829587.640099:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973164160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829587.640105:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973164288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829587.640123:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829587.640136:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829587/real 1787829587] req@ffff9c1a78861680 x1874668973164288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829587.640147:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973164288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829587.640162:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829587.640165:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829587.640168:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973164288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829588.151482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829588.151482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829588.407555:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829589.175422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829589.175437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829589.431602:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829590.199454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829590.199454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829590.455537:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829591.223439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829591.223467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829591.479509:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829592.247464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829592.247464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829592.503511:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829592.759460:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829592.759470:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829592.759475:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829592.759485:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6043 00000100:00080000:2.0:1787829592.759502:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829592.759510:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829592.759550:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973164416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829592.759576:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829592.759580:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829592.759582:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829592.759587:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6043 00000100:00080000:2.0:1787829592.759592:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829592.759597:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829592.759607:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829592.759612:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973164416:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829592.759624:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973164544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829592.759638:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829592.759642:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829592.759644:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829592.759648:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6043 00000100:00080000:2.0:1787829592.759653:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829592.759656:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829592.759661:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829592.759672:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829592.759676:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973164672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829592.759705:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829592.759709:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829592.759711:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829592.759716:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6043 00000100:00080000:2.0:1787829592.759720:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829592.759725:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829592.759730:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829592.759743:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829592/real 1787829592] req@ffff9c1a49d8e1c0 x1874668973164416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829592.759745:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973164800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829592.759782:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973164416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829592.759783:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829592.759791:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829592.759796:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829592.759799:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829592.759803:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6043 00000100:00080000:1.0:1787829592.759807:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829592.759810:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829592.759813:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973164416:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829592.759819:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829592.759826:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829592.759837:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973164544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829592.759842:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973164928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829592.759856:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829592.759860:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787829592.759863:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829592.759865:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829592.759872:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6043 00000100:00080000:2.0:1787829592.759876:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829592.759878:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829592/real 1787829592] req@ffff9c1a49d8cf00 x1874668973164544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829592.759883:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829592.759895:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829592.759898:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973164544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829592.759907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829592.759913:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829592.759914:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973165056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829592.759917:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973164544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829592.759929:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973164672:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829592.759935:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829592.759940:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829592.759942:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829592.759949:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6043 00000800:00000100:1.0:1787829592.759950:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829592.759957:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829592.759963:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829592/real 1787829592] req@ffff9c1a49d8c000 x1874668973164672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829592.759966:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829592.759977:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829592.759981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973164672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829592.759988:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829592.759991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829592.759996:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973165184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829592.759997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973164672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829592.760009:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973164800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829592.760031:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829592.760039:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829592/real 1787829592] req@ffff9c1a49d8c780 x1874668973164800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829592.760047:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973164800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829592.760057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829592.760059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829592.760061:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973164800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829592.760071:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973164928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829592.760081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829592.760087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829592/real 1787829592] req@ffff9c1a49d8de00 x1874668973164928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829592.760094:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973164928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829592.760099:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829592.760101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829592.760102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973164928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829592.760106:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973165056:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829592.760116:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829592.760122:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829592/real 1787829592] req@ffff9c1a49d8da40 x1874668973165056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829592.760129:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973165056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829592.760134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829592.760136:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829592.760138:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973165056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829592.760142:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973165184:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829592.760150:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829592.760156:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829592/real 1787829592] req@ffff9c1a49d8cb40 x1874668973165184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829592.760162:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973165184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829592.760168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829592.760169:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829592.760171:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973165184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829593.271447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829593.271448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829593.527550:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829594.295455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829594.295455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829594.551507:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829595.319446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829595.319446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829595.575487:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829596.343482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829596.343486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829596.599590:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829597.367450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829597.367450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829597.623542:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829597.879457:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829597.879467:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829597.879472:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829597.879482:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6048 00000100:00080000:2.0:1787829597.879500:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829597.879508:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829597.879546:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973165312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829597.879576:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829597.879579:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829597.879581:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829597.879586:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6048 00000100:00080000:2.0:1787829597.879590:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829597.879594:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829597.879604:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829597.879617:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973165440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829597.879622:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973165312:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829597.879637:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829597.879643:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829597.879645:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829597.879664:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6048 00000100:00080000:2.0:1787829597.879669:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829597.879673:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829597.879678:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829597.879691:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973165568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829597.879702:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829597.879703:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829597.879709:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829597.879712:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829597.879716:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6048 00000100:00080000:2.0:1787829597.879723:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829597.879727:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829597.879732:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829597.879744:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973165696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829597.879747:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829597/real 1787829597] req@ffff9c1a49d8cb40 x1874668973165312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829597.879768:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829597.879773:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829597.879776:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829597.879777:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973165312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829597.879785:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6048 00000100:00080000:1.0:1787829597.879793:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829597.879794:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829597.879800:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829597.879802:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829597.879806:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973165312:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787829597.879811:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829597.879826:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973165824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829597.879836:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973165440:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829597.879839:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829597.879846:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829597.879848:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829597.879854:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6048 00000100:00080000:2.0:1787829597.879858:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829597.879862:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829597.879868:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829597.879872:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829597.879885:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973165952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829597.879892:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829597/real 1787829597] req@ffff9c1a49d8da40 x1874668973165440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829597.879899:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829597.879908:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829597.879913:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829597.879920:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6048 00000100:00100000:1.0:1787829597.879921:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973165440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829597.879926:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829597.879934:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787829597.879939:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829597.879941:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829597.879946:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829597.879952:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973165440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829597.879957:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973166080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829597.879964:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973165568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829597.879996:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829597.880009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829597/real 1787829597] req@ffff9c1a49d8de00 x1874668973165568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829597.880018:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973165568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829597.880024:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829597.880026:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829597.880028:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973165568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829597.880033:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973165696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829597.880049:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829597.880060:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829597/real 1787829597] req@ffff9c1a49d8c780 x1874668973165696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829597.880073:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973165696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829597.880082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829597.880085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829597.880089:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973165696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829597.880105:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973165824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829597.880121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829597.880129:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829597/real 1787829597] req@ffff9c1a49d8c000 x1874668973165824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829597.880136:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973165824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829597.880142:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829597.880144:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829597.880146:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973165824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829597.880149:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973165952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829597.880160:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829597.880171:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829597/real 1787829597] req@ffff9c1a49d8cf00 x1874668973165952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829597.880183:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973165952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829597.880192:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829597.880196:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829597.880199:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973165952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829597.880206:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973166080:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829597.880225:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829597.880233:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829597/real 1787829597] req@ffff9c1a49d8e1c0 x1874668973166080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829597.880241:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973166080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829597.880247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829597.880249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829597.880251:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973166080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829598.391464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829598.391464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829598.647547:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829599.415457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829599.415462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829599.671541:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829600.439465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829600.439474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829600.695527:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829601.463433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829601.463433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829601.719492:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829602.487434:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829602.487434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829602.743491:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829602.999468:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829602.999478:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829602.999483:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829602.999494:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6053 00000100:00080000:0.0:1787829602.999506:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829602.999513:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829602.999550:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973166208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829602.999587:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829602.999592:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829602.999595:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829602.999601:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6053 00000100:00080000:0.0:1787829602.999607:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829602.999627:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829602.999628:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973166208:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829602.999641:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829602.999662:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973166336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829602.999675:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829602.999679:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829602.999681:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829602.999685:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829602.999688:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6053 00000100:00080000:0.0:1787829602.999696:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829602.999700:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829602.999706:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829602.999718:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973166464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829602.999729:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829602.999733:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829602.999736:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829602.999741:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6053 00000100:00080000:0.0:1787829602.999746:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829602.999752:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829602.999758:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829602.999763:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829602/real 1787829602] req@ffff9c1a78860000 x1874668973166208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829602.999774:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973166592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829602.999783:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973166208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829602.999793:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829602.999796:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829602.999798:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829602.999801:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829602.999804:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829602.999807:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973166208:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829602.999813:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6053 00000100:00080000:0.0:1787829602.999819:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829602.999824:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829602.999830:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829602.999831:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973166336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829602.999851:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973166720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829602.999860:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829602.999872:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829602.999877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829602/real 1787829602] req@ffff9c1a78861680 x1874668973166336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829602.999878:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829602.999882:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829602.999891:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6053 00000100:00080000:0.0:1787829602.999900:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829602.999904:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973166336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829602.999909:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829602.999921:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829602.999923:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829602.999931:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829602.999936:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973166336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829602.999940:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973166848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829602.999949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973166464:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829602.999965:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829602.999970:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829602.999973:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829602.999977:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829602.999981:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6053 00000100:00080000:0.0:1787829602.999987:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829602.999991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829602/real 1787829602] req@ffff9c1a78862580 x1874668973166464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829602.999994:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829603.000003:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829603.000008:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973166464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829603.000018:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973166976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829603.000052:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829603.000056:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829603.000060:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973166464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829603.000069:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973166592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829603.000092:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829603.000104:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829602/real 1787829603] req@ffff9c1a788630c0 x1874668973166592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829603.000118:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973166592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829603.000124:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829603.000127:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829603.000129:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973166592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829603.000143:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973166720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829603.000158:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829603.000166:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829602/real 1787829603] req@ffff9c1a78863c00 x1874668973166720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829603.000173:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973166720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829603.000179:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829603.000181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829603.000183:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973166720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829603.000187:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973166848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829603.000198:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829603.000209:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829602/real 1787829603] req@ffff9c1a788612c0 x1874668973166848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829603.000217:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973166848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829603.000222:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829603.000224:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829603.000226:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973166848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829603.000230:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973166976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829603.000239:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829603.000245:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829602/real 1787829603] req@ffff9c1a78861e00 x1874668973166976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829603.000253:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973166976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829603.000258:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829603.000259:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829603.000261:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973166976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829603.511485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829603.511495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829603.767514:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829604.535414:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829604.535421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829604.791530:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829605.559419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829605.559507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829605.815497:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829606.583415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829606.583415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829606.839499:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829607.607445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829607.607445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829607.863519:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829608.119452:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829608.119463:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829608.119467:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829608.119478:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6058 00000100:00080000:0.0:1787829608.119489:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829608.119497:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829608.119544:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973167104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829608.119575:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829608.119596:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829608.119598:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829608.119603:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6058 00000100:00080000:0.0:1787829608.119608:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829608.119614:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829608.119625:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829608.119641:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973167232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829608.119647:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973167104:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829608.119664:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829608.119668:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829608.119670:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829608.119676:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6058 00000100:00080000:0.0:1787829608.119681:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829608.119686:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829608.119692:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829608.119706:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973167360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829608.119716:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829608.119723:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829608.119729:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829608.119731:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829608.119736:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6058 00000100:00080000:0.0:1787829608.119741:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829608.119745:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829608.119752:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829608.119763:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff9c1a78860780 x1874668973167104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829608.119767:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973167488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829608.119796:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829608.119800:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829608.119804:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829608.119804:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973167104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829608.119814:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6058 00000100:00080000:0.0:1787829608.119822:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829608.119826:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829608.119828:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829608.119835:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787829608.119837:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829608.119844:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973167104:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829608.119855:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973167616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829608.119866:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829608.119869:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829608.119872:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829608.119875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973167232:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829608.119880:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6058 00000100:00080000:0.0:1787829608.119886:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829608.119891:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829608.119897:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829608.119914:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973167744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829608.119914:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829608.119930:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829608.119935:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829608.119935:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff9c1a78862940 x1874668973167232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829608.119940:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829608.119949:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6058 00000100:00080000:0.0:1787829608.119956:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829608.119961:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973167232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829608.119965:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829608.119974:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829608.119977:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829608.119984:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829608.119990:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973167872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829608.119993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973167232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829608.120008:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973167360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829608.120039:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829608.120054:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff9c1a78861a40 x1874668973167360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829608.120067:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973167360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829608.120075:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829608.120079:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829608.120082:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973167360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829608.120088:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973167488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829608.120107:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829608.120118:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff9c1a78860f00 x1874668973167488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829608.120130:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973167488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829608.120139:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829608.120142:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829608.120146:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973167488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829608.120151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973167616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829608.120169:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829608.120179:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff9c1a788621c0 x1874668973167616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829608.120197:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973167616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829608.120215:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829608.120219:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829608.120222:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973167616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829608.120242:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973167744:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829608.120263:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829608.120274:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff9c1a78863840 x1874668973167744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829608.120286:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973167744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829608.120294:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829608.120298:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829608.120301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973167744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829608.120307:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973167872:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829608.120326:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829608.120338:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829608/real 1787829608] req@ffff9c1a78862d00 x1874668973167872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829608.120388:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973167872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829608.120403:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829608.120407:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829608.120411:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973167872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829608.631403:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829608.631404:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829608.887511:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829609.655460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829609.655462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829609.911517:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829610.679452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829610.679476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829610.935491:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829611.703476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829611.703502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829611.959532:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829612.727443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829612.727443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829612.983473:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829613.239470:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829613.239481:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829613.239485:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.239495:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6063 00000100:00080000:1.0:1787829613.239507:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829613.239515:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829613.239565:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973168000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829613.239593:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829613.239597:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829613.239600:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.239604:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6063 00000100:00080000:1.0:1787829613.239610:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829613.239616:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829613.239627:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829613.239628:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973168000:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829613.239647:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973168128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829613.239661:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829613.239665:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829613.239668:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.239672:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6063 00000100:00080000:1.0:1787829613.239678:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829613.239682:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829613.239686:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:1.0:1787829613.239692:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829613.239708:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973168256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829613.239719:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829613.239726:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829613.239726:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff9c1a78a7bc00 x1874668973168000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829613.239731:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.239742:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6063 00000100:00080000:1.0:1787829613.239748:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829613.239753:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973168000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829613.239757:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829613.239767:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829613.239772:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829613.239780:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829613.239784:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973168384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829613.239789:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973168000:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829613.239811:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829613.239814:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829613.239817:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973168128:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829613.239819:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.239826:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6063 00000100:00080000:1.0:1787829613.239831:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829613.239836:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829613.239842:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:1.0:1787829613.239843:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829613.239860:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973168512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829613.239865:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff9c1a78a78780 x1874668973168128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829613.239891:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829613.239894:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973168128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829613.239899:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829613.239904:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.239914:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6063 00000100:00080000:2.0:1787829613.239915:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829613.239920:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829613.239921:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829613.239927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973168128:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829613.239932:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829613.239936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973168256:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787829613.239943:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829613.239954:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829613.239962:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973168640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829613.239966:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff9c1a78a7a940 x1874668973168256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829613.239995:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829613.239995:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973168256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829613.240004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829613.240007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829613.240009:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973168256:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829613.240016:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829613.240016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973168384:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829613.240022:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829613.240030:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6063 00000100:00080000:1.0:1787829613.240035:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829613.240035:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829613.240043:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829613.240045:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff9c1a78a79e00 x1874668973168384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829613.240053:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829613.240057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973168384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829613.240064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829613.240067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829613.240072:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973168768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829613.240072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973168384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829613.240093:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973168512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829613.240106:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829613.240116:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff9c1a78a783c0 x1874668973168512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829613.240124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973168512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829613.240129:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829613.240131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829613.240133:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973168512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829613.240137:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973168640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829613.240147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829613.240153:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff9c1a78a7a1c0 x1874668973168640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829613.240160:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973168640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829613.240165:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829613.240167:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829613.240169:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973168640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829613.240172:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973168768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829613.240181:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829613.240187:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829613/real 1787829613] req@ffff9c1a78a7b0c0 x1874668973168768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829613.240195:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973168768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829613.240200:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829613.240201:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829613.240203:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973168768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829613.751448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829613.751449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829614.007497:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829614.775443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829614.775443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829615.031483:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829615.799454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829615.799454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829616.055497:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829616.823446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829616.823446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829617.079467:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829617.847448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829617.847448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829618.103505:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829618.359554:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829618.359565:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829618.359569:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829618.359580:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6068 00000100:00080000:3.0:1787829618.359593:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829618.359600:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829618.359644:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973168896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829618.359678:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829618.359681:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829618.359683:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829618.359687:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6068 00000100:00080000:3.0:1787829618.359689:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829618.359693:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829618.359704:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829618.359716:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973169024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829618.359727:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829618.359733:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829618.359740:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829618.359741:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973168896:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829618.359751:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6068 00000100:00080000:3.0:1787829618.359758:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829618.359764:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829618.359779:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829618.359798:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973169152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829618.359808:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829618.359812:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829618.359819:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829618.359824:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829618.359827:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6068 00000100:00080000:3.0:1787829618.359833:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829618.359839:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829618.359845:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829618.359854:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973169280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829618.359868:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829618.359873:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829618.359876:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff9c1a7895de00 x1874668973168896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829618.359876:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829618.359888:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6068 00000100:00080000:3.0:1787829618.359897:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829618.359910:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829618.359914:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829618.359920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973168896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829618.359931:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973169408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829618.359937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829618.359943:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829618.359948:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829618.359953:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829618.359959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973168896:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829618.359961:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829618.359972:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6068 00000100:00080000:3.0:1787829618.359978:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829618.359988:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829618.359990:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973169024:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829618.360013:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829618.360036:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829618.360036:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973169536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829618.360055:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff9c1a7895f0c0 x1874668973169024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829618.360055:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829618.360075:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829618.360084:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829618.360102:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6068 00000100:00100000:2.0:1787829618.360107:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973169024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829618.360110:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829618.360128:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829618.360146:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829618.360150:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829618.360156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829618.360169:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973169024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829618.360169:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973169664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829618.360192:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973169152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829618.360221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829618.360231:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff9c1a7895f840 x1874668973169152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829618.360240:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973169152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829618.360246:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829618.360251:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829618.360254:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973169152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829618.360259:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973169280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829618.360269:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829618.360275:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff9c1a7895fc00 x1874668973169280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829618.360282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973169280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829618.360287:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829618.360289:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829618.360291:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973169280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829618.360295:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973169408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829618.360305:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829618.360310:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff9c1a7895e1c0 x1874668973169408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829618.360317:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973169408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829618.360323:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829618.360324:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829618.360326:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973169408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829618.360339:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973169536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829618.360375:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829618.360383:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff9c1a7895da40 x1874668973169536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829618.360391:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973169536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829618.360397:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829618.360399:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829618.360401:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973169536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829618.360405:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973169664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829618.360416:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829618.360422:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829618/real 1787829618] req@ffff9c1a7895cb40 x1874668973169664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829618.360429:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973169664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829618.360434:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829618.360436:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829618.360438:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973169664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829618.871419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829618.871421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829619.127500:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829619.895455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829619.895462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829620.151653:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829620.919419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829620.919446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829621.175468:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829621.943427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829621.943427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829622.199585:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829622.967408:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829622.967408:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829623.223483:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829623.479454:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829623.479462:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829623.479466:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829623.479487:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6073 00000100:00080000:3.0:1787829623.479497:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829623.479503:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829623.479535:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973169792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829623.479558:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829623.479560:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829623.479562:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829623.479565:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6073 00000100:00080000:3.0:1787829623.479567:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829623.479569:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829623.479576:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829623.479585:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973169920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829623.479593:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829623.479595:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829623.479597:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829623.479603:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973169792:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829623.479603:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6073 00000100:00080000:3.0:1787829623.479610:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829623.479613:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829623.479616:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829623.479625:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973170048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829623.479637:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829623.479640:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829623.479641:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829623.479644:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6073 00000100:00080000:3.0:1787829623.479659:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829623.479662:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829623.479664:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829623.479672:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829623.479681:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973170176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829623.479694:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829623.479697:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829623.479698:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829623.479701:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6073 00000100:00080000:3.0:1787829623.479705:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829623.479707:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829623.479711:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829623.479720:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973170304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829623.479726:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829623.479728:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829623.479730:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829623.479735:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829623/real 1787829623] req@ffff9c1a7895d2c0 x1874668973169792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829623.479735:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6073 00000100:00080000:3.0:1787829623.479741:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829623.479746:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829623.479753:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829623.479766:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973170432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829623.479773:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829623.479774:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973169792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829623.479775:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829623.479779:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829623.479786:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6073 00000100:00080000:3.0:1787829623.479791:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829623.479794:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829623.479795:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829623.479802:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829623.479804:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829623.479810:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973169792:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829623.479814:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973170560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829623.479836:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973169920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829623.479859:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829623.479870:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829623/real 1787829623] req@ffff9c1a7895f480 x1874668973169920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829623.479878:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973169920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829623.479889:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829623.479891:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829623.479894:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973169920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829623.479898:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973170048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829623.479910:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829623.479916:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829623/real 1787829623] req@ffff9c1a7895c000 x1874668973170048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829623.479923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973170048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829623.479928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829623.479930:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829623.479932:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973170048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829623.479936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973170176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829623.479946:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829623.479951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829623/real 1787829623] req@ffff9c1a7895e580 x1874668973170176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829623.479959:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973170176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829623.479964:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829623.479966:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829623.479968:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973170176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829623.479972:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973170304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829623.479982:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829623.479987:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829623/real 1787829623] req@ffff9c1a7895cf00 x1874668973170304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829623.479994:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973170304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829623.479999:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829623.480001:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829623.480003:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973170304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829623.480007:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973170432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829623.480018:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829623.480024:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829623/real 1787829623] req@ffff9c1a7895e940 x1874668973170432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829623.480031:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973170432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829623.480036:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829623.480038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829623.480040:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973170432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829623.480043:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973170560:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829623.480052:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829623.480058:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829623/real 1787829623] req@ffff9c1a7895c780 x1874668973170560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829623.480065:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973170560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829623.480071:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829623.480072:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829623.480074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973170560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829623.991424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829623.991424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829624.247515:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829625.015442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829625.015442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829625.271484:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829626.039436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829626.039447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829626.295598:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829627.063430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829627.063441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829627.319573:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829628.087426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829628.087426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829628.343560:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829628.599486:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829628.599497:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829628.599502:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829628.599513:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6078 00000100:00080000:0.0:1787829628.599531:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829628.599539:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829628.599582:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973170688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829628.599606:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829628.599611:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829628.599613:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829628.599616:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6078 00000100:00080000:0.0:1787829628.599619:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829628.599622:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829628.599631:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829628.599644:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973170816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829628.599654:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829628.599654:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973170688:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829628.599662:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829628.599664:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829628.599668:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6078 00000100:00080000:0.0:1787829628.599670:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829628.599673:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829628.599675:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829628.599684:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973170944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829628.599692:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829628.599695:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829628.599697:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829628.599706:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6078 00000100:00080000:0.0:1787829628.599708:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829628.599711:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829628.599715:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829628.599723:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973171072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829628.599730:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829628.599732:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829628.599734:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829628.599739:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829628.599741:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6078 00000100:00080000:0.0:1787829628.599745:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829628.599747:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829628.599750:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829628.599758:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973171200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829628.599765:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829628.599767:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829628.599769:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829628.599771:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6078 00000100:00080000:0.0:1787829628.599773:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829628.599776:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829628.599779:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829628.599788:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973171328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829628.599798:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829628.599801:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829628/real 1787829628] req@ffff9c1a78860b40 x1874668973170688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829628.599803:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829628.599807:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829628.599816:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6078 00000100:00080000:0.0:1787829628.599824:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829628.599829:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829628.599835:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829628.599837:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973170688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829628.599848:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973171456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829628.599851:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829628.599861:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829628.599865:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973170688:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829628.599892:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973170816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829628.599918:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829628.599929:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829628/real 1787829628] req@ffff9c1a78862d00 x1874668973170816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829628.599937:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973170816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829628.599948:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829628.599951:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829628.599953:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973170816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829628.599957:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973170944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829628.599969:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829628.599975:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829628/real 1787829628] req@ffff9c1a78863840 x1874668973170944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829628.599983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973170944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829628.599988:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829628.599990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829628.599992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973170944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829628.599995:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973171072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829628.600006:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829628.600012:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829628/real 1787829628] req@ffff9c1a788621c0 x1874668973171072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829628.600019:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973171072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829628.600025:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829628.600026:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829628.600029:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973171072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829628.600033:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973171200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829628.600042:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829628.600048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829628/real 1787829628] req@ffff9c1a78860f00 x1874668973171200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829628.600055:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973171200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829628.600060:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829628.600062:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829628.600065:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973171200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829628.600071:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973171328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829628.600089:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829628.600101:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829628/real 1787829628] req@ffff9c1a78861a40 x1874668973171328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829628.600115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973171328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829628.600124:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829628.600126:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829628.600128:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973171328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829628.600133:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973171456:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829628.600148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829628.600156:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829628/real 1787829628] req@ffff9c1a78862940 x1874668973171456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829628.600164:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973171456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829628.600169:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829628.600171:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829628.600173:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973171456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829629.111488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829629.111488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829629.367567:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829630.135415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829630.135415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829630.391539:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829631.159453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829631.159454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829631.415588:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829632.183500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829632.183509:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829632.439533:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829633.207486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829633.207486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829633.463742:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829633.719473:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829633.719484:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829633.719489:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829633.719499:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6084 00000100:00080000:2.0:1787829633.719511:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829633.719518:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829633.719571:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973171584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829633.719602:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829633.719605:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829633.719607:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829633.719610:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6084 00000100:00080000:2.0:1787829633.719613:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829633.719615:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829633.719627:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829633.719637:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973171712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829633.719647:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829633.719650:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829633.719651:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829633.719661:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973171584:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829633.719661:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6084 00000100:00080000:2.0:1787829633.719677:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829633.719685:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829633.719693:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829633.719710:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973171840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829633.719723:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829633.719727:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829633.719729:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829633.719732:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6084 00000100:00080000:2.0:1787829633.719737:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829633.719741:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829633.719746:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829633.719758:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973171968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829633.719767:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829633.719769:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829633.719771:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829633.719774:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6084 00000100:00080000:2.0:1787829633.719779:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829633.719783:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829633.719787:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973171712:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787829633.719790:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829633.719807:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973172096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829633.719816:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829633.719819:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829633.719821:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829633.719826:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973171840:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829633.719832:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6084 00000100:00080000:2.0:1787829633.719850:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829633.719856:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829633.719863:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829633.719878:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973171968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829633.719880:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973172224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829633.719902:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829633.719906:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829633.719909:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829633.719916:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6084 00000100:00080000:2.0:1787829633.719919:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829633.719927:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973172096:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829633.719930:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829633.719937:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829633.719954:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973172352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829633.719982:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973172224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829633.720010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973172352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829634.231481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829634.231512:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829634.487537:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829635.255468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829635.255468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829635.511593:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829636.279450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829636.279467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829636.535534:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829636.791665:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787829636.791715:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829636.791732:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829636.791770:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829636.791777:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829636.791784:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829636.791792:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829636.791799:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829636.791807:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787829636.791824:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829633/real 1787829636] req@ffff9c1a49d8fc00 x1874668973172096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829636.791857:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973172096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829636.791879:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829636.791886:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829636.791891:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973172096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829636.791899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829633/real 1787829636] req@ffff9c1a49d8ed00 x1874668973172224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829636.791913:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973172224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829636.791921:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829636.791925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829636.791929:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973172224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829636.791936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829633/real 1787829636] req@ffff9c1a49d8f0c0 x1874668973171712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829636.791947:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973171712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829636.791955:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829636.791958:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829636.791961:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973171712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829636.791968:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829633/real 1787829636] req@ffff9c1a49d8c3c0 x1874668973171840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829636.791980:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973171840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829636.791988:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829636.791991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829636.791995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973171840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829636.792002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829633/real 1787829636] req@ffff9c1a49d8e580 x1874668973171968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829636.792014:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973171968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829636.792023:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829636.792027:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829636.792031:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973171968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829636.792038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829633/real 1787829636] req@ffff9c1a49d8d680 x1874668973171584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829636.792056:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973171584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829636.792065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829636.792069:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829636.792072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973171584:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829636.792110:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829633/real 1787829636] req@ffff9c1a49d8e1c0 x1874668973172352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829636.792123:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973172352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829636.792133:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829636.792137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829636.792141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973172352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829637.303454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829637.303477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829637.559484:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829638.327427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829638.327453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829638.583555:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829638.839419:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829638.839430:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829638.839436:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829638.839447:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6089 00000100:00080000:2.0:1787829638.839459:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829638.839466:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829638.839506:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973172480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829638.839539:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829638.839544:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829638.839545:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829638.839549:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6089 00000100:00080000:2.0:1787829638.839553:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829638.839556:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829638.839566:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829638.839583:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973172608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829638.839593:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829638.839596:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829638.839599:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829638.839602:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6089 00000100:00080000:2.0:1787829638.839604:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829638.839607:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829638.839614:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829638.839619:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973172480:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829638.839630:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973172736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829638.839643:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829638.839647:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829638.839651:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829638.839656:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6089 00000100:00080000:2.0:1787829638.839663:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829638.839669:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829638.839676:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829638.839691:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973172864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829638.839702:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829638.839709:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829638.839711:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829638.839718:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829638.839725:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6089 00000100:00080000:2.0:1787829638.839733:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829638.839742:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829638.839749:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829638.839768:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973172992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829638.839783:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829638.839785:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829638/real 1787829638] req@ffff9c1a49d8cf00 x1874668973172480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829638.839789:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829638.839794:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829638.839812:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6089 00000100:00080000:2.0:1787829638.839822:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829638.839831:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829638.839840:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973172480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829638.839841:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829638.839868:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829638.839869:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973173120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829638.839880:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829638.839899:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973172480:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829638.839907:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829638.839913:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829638.839917:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829638.839920:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973172608:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829638.839926:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6089 00000100:00080000:2.0:1787829638.839934:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829638.839941:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829638.839948:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829638.839964:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829638.839970:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973173248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829638.839992:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829638/real 1787829638] req@ffff9c1a49d8c000 x1874668973172608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829638.840011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973172608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829638.840022:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829638.840026:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829638.840030:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973172608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829638.840062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973172736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829638.840087:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829638.840101:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829638/real 1787829638] req@ffff9c1a49d8c780 x1874668973172736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829638.840112:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973172736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829638.840120:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829638.840123:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829638.840126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973172736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829638.840134:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973172864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829638.840153:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829638.840165:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829638/real 1787829638] req@ffff9c1a49d8de00 x1874668973172864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829638.840178:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973172864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829638.840187:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829638.840191:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829638.840194:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973172864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829638.840213:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973172992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829638.840646:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829638.840665:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829638/real 1787829638] req@ffff9c1a49d8da40 x1874668973172992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829638.840678:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973172992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829638.840687:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829638.840691:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829638.840694:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973172992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829638.840702:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973173120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829638.840727:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829638.840738:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829638/real 1787829638] req@ffff9c1a49d8cb40 x1874668973173120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829638.840752:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973173120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829638.840761:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829638.840764:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829638.840768:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973173120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829638.840775:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973173248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829638.840802:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829638.840816:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829638/real 1787829638] req@ffff9c1a49d8e1c0 x1874668973173248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829638.840829:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973173248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829638.840838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829638.840842:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829638.840845:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973173248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829639.351471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829639.351471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829639.607580:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829640.375431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829640.375431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829640.631538:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829641.399455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829641.399455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829641.655590:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829642.423449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829642.423449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829642.679535:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829643.447443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829643.447444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829643.703551:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829643.959485:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829643.959496:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829643.959501:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829643.959511:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6094 00000100:00080000:0.0:1787829643.959523:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829643.959530:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829643.959571:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973173376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829643.959603:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829643.959608:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829643.959610:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829643.959615:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6094 00000100:00080000:0.0:1787829643.959620:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829643.959625:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829643.959636:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829643.959651:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973173376:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829643.959653:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973173504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829643.959672:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829643.959676:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829643.959678:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829643.959683:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6094 00000100:00080000:0.0:1787829643.959687:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829643.959693:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829643.959700:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829643.959713:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973173632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829643.959725:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829643.959729:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829643.959734:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829643.959737:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829643.959743:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6094 00000100:00080000:0.0:1787829643.959747:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829643.959752:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829643.959758:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829643.959771:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973173760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829643.959776:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829643/real 1787829643] req@ffff9c1a78860780 x1874668973173376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829643.959793:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829643.959799:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829643.959801:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829643.959807:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6094 00000100:00100000:3.0:1787829643.959808:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973173376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829643.959813:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829643.959822:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:3.0:1787829643.959827:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829643.959829:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829643.959836:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829643.959843:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973173376:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829643.959846:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973173888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829643.959860:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829643.959864:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829643.959866:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829643.959871:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6094 00000100:00100000:3.0:1787829643.959874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973173504:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829643.959877:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829643.959886:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829643.959892:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829643.959908:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973174016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829643.959915:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829643.959929:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829643.959933:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829643.959936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829643/real 1787829643] req@ffff9c1a78861e00 x1874668973173504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829643.959937:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829643.959946:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6094 00000100:00080000:0.0:1787829643.959954:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829643.959962:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829643.959972:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829643.959980:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973173504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829643.959989:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973174144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829643.959994:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829643.960002:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829643.960006:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973173504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829643.960018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973173632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829643.960042:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829643.960056:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829643/real 1787829643] req@ffff9c1a788612c0 x1874668973173632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829643.960069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973173632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829643.960077:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829643.960081:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829643.960084:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973173632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829643.960090:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973173760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829643.960119:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829643.960132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829643/real 1787829643] req@ffff9c1a78863c00 x1874668973173760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829643.960144:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973173760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829643.960159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829643.960162:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829643.960166:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973173760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829643.960171:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973173888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829643.960189:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829643.960198:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829643/real 1787829643] req@ffff9c1a788630c0 x1874668973173888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829643.960209:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973173888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829643.960217:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829643.960220:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829643.960223:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973173888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829643.960240:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973174016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829643.960258:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829643.960267:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829643/real 1787829643] req@ffff9c1a78862580 x1874668973174016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829643.960279:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973174016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829643.960287:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829643.960290:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829643.960293:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973174016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829643.960299:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973174144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829643.960314:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829643.960323:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829643/real 1787829643] req@ffff9c1a78861680 x1874668973174144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829643.960373:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973174144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829643.960381:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829643.960384:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829643.960387:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973174144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829644.471482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829644.471487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829644.727533:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829645.495409:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829645.495409:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829645.751551:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829646.519446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829646.519446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829646.775496:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829647.543417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829647.543417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829647.799510:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829648.567403:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829648.567411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829648.823586:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829649.079472:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829649.079483:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829649.079488:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829649.079499:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6099 00000100:00080000:3.0:1787829649.079512:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829649.079518:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829649.079562:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973174272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829649.079593:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829649.079596:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829649.079601:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829649.079605:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6099 00000100:00080000:3.0:1787829649.079607:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829649.079611:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829649.079622:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829649.079634:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973174400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829649.079643:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829649.079645:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973174272:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829649.079647:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829649.079651:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829649.079656:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6099 00000100:00080000:3.0:1787829649.079659:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829649.079661:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829649.079664:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829649.079675:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973174528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829649.079683:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829649.079686:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829649.079687:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829649.079689:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6099 00000100:00080000:3.0:1787829649.079692:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829649.079695:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829649.079699:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829649.079707:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973174656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829649.079713:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829649.079715:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829649.079717:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829649.079719:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6099 00000100:00080000:3.0:1787829649.079722:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829649.079728:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829649.079728:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829649.079733:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829649.079742:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973174784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829649.079753:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829649.079757:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829649.079759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829649.079763:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6099 00000100:00080000:3.0:1787829649.079768:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829649.079774:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829649.079781:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829649.079782:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff9c1a7895c3c0 x1874668973174272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829649.079800:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973174912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829649.079804:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973174272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829649.079831:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829649.079833:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829649.079836:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829649.079839:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829649.079843:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973174272:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829649.079844:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829649.079852:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6099 00000100:00080000:3.0:1787829649.079856:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829649.079859:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829649.079862:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829649.079870:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973175040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829649.079878:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973174400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829649.079908:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829649.079924:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff9c1a7895c780 x1874668973174400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829649.079936:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973174400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829649.079943:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829649.079945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829649.079948:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973174400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829649.079953:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973174528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829649.079969:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829649.079981:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff9c1a7895e940 x1874668973174528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829649.079994:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973174528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829649.080003:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829649.080007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829649.080011:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973174528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829649.080018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973174656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829649.080039:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829649.080052:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff9c1a7895cf00 x1874668973174656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829649.080065:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973174656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829649.080083:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829649.080086:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829649.080089:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973174656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829649.080093:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973174784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829649.080110:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829649.080118:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff9c1a7895e580 x1874668973174784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829649.080126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973174784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829649.080132:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829649.080135:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829649.080138:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973174784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829649.080144:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973174912:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829649.080165:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829649.080178:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff9c1a7895c000 x1874668973174912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829649.080191:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973174912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829649.080197:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829649.080199:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829649.080203:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973174912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829649.080217:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973175040:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829649.080233:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829649.080240:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829649/real 1787829649] req@ffff9c1a7895f480 x1874668973175040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829649.080248:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973175040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829649.080254:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829649.080256:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829649.080258:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973175040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829649.591402:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829649.591407:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829649.847518:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829650.615448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829650.615448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829650.871573:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829651.639468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829651.639468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829651.895488:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829652.663489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829652.663489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829652.919585:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829653.687452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829653.687452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829653.943496:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829654.199470:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829654.199481:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829654.199486:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829654.199495:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6104 00000100:00080000:0.0:1787829654.199508:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829654.199516:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829654.199551:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973175168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829654.199576:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829654.199580:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829654.199582:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829654.199587:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6104 00000100:00080000:0.0:1787829654.199591:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829654.199596:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829654.199606:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829654.199623:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973175296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829654.199631:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829654.199634:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829654.199635:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829654.199638:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6104 00000100:00080000:0.0:1787829654.199641:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829654.199644:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829654.199648:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829654.199656:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973175424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829654.199665:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829654.199672:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829654.199674:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829654.199675:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973175168:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829654.199677:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6104 00000100:00080000:0.0:1787829654.199680:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829654.199693:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829654.199700:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829654.199719:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973175552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829654.199727:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829654.199728:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829654.199730:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829654.199732:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6104 00000100:00080000:0.0:1787829654.199734:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829654.199737:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829654.199740:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829654.199747:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973175680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829654.199753:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829654.199755:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829654.199757:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829654.199759:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6104 00000100:00080000:0.0:1787829654.199764:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829654.199767:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829654.199771:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829654.199777:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973175808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829654.199790:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829654.199796:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787829654.199800:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829654.199801:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829654.199811:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6104 00000100:00080000:0.0:1787829654.199817:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829654.199823:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829654.199828:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829654.199843:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973175936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829654.199873:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff9c1a78860000 x1874668973175168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829654.199898:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973175168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.199911:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829654.199916:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829654.199922:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973175168:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829654.199931:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973175296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829654.199958:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829654.199971:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff9c1a78861680 x1874668973175296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829654.199985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973175296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.199994:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829654.199997:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829654.200001:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973175296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829654.200031:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973175424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829654.200054:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829654.200065:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff9c1a78862580 x1874668973175424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829654.200077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973175424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.200085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829654.200088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829654.200091:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973175424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829654.200098:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973175552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829654.200119:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829654.200132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff9c1a788630c0 x1874668973175552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829654.200144:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973175552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.200154:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829654.200157:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829654.200161:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973175552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829654.200167:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973175680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829654.200187:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829654.200199:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff9c1a78863c00 x1874668973175680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829654.200212:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973175680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.200220:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829654.200224:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829654.200228:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973175680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829654.200234:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973175808:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829654.200253:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829654.200284:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff9c1a788612c0 x1874668973175808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829654.200296:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973175808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.200305:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829654.200309:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829654.200313:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973175808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829654.200319:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973175936:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829654.200338:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829654.200397:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829654/real 1787829654] req@ffff9c1a78861e00 x1874668973175936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829654.200409:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973175936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829654.200418:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829654.200422:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829654.200425:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973175936:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829654.711450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829654.711450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829654.967538:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829655.735423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829655.735423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829655.991487:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829656.759451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829656.759451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829657.015496:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829657.783442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829657.783443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829658.039489:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829658.807452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829658.807456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829659.063551:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829659.319450:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829659.319461:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829659.319466:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829659.319477:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6109 00000100:00080000:2.0:1787829659.319489:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829659.319497:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829659.319540:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973176064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829659.319569:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829659.319573:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829659.319575:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829659.319581:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6109 00000100:00080000:2.0:1787829659.319585:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829659.319589:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829659.319602:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829659.319618:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973176192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829659.319628:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829659.319631:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829659.319633:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829659.319637:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6109 00000100:00080000:2.0:1787829659.319641:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829659.319643:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829659.319647:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829659.319657:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973176320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829659.319668:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829659.319673:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973176064:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829659.319673:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829659.319677:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829659.319689:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6109 00000100:00080000:2.0:1787829659.319696:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829659.319702:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829659.319709:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829659.319724:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973176448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829659.319732:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829659.319734:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829659.319736:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829659.319739:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6109 00000100:00080000:2.0:1787829659.319745:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829659.319748:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829659.319751:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829659.319759:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973176576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829659.319765:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829659.319767:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829659.319770:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829659.319776:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829659.319777:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6109 00000100:00080000:2.0:1787829659.319785:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829659.319789:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829659.319793:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829659.319801:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973176704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829659.319812:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829659.319815:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829659.319816:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829659.319819:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6109 00000100:00080000:2.0:1787829659.319822:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829659.319825:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829659.319828:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829659.319841:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973176832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829659.319842:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff9c1a49d8d680 x1874668973176064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829659.319878:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973176064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829659.319900:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829659.319905:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829659.319910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973176064:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829659.319919:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973176192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829659.319948:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829659.319962:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff9c1a49d8e580 x1874668973176192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829659.319976:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973176192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829659.319986:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829659.319990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829659.319994:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973176192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829659.320020:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973176320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829659.320041:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829659.320053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff9c1a49d8c3c0 x1874668973176320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829659.320076:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973176320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829659.320085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829659.320090:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829659.320094:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973176320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829659.320101:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973176448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829659.320123:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829659.320135:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff9c1a49d8f0c0 x1874668973176448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829659.320149:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973176448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829659.320159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829659.320163:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829659.320166:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973176448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829659.320173:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973176576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829659.320191:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829659.320202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff9c1a49d8ed00 x1874668973176576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829659.320215:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973176576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829659.320240:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829659.320244:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829659.320248:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973176576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829659.320255:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973176704:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829659.320277:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829659.320289:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff9c1a49d8fc00 x1874668973176704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829659.320303:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973176704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829659.320312:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829659.320316:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829659.320320:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973176704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829659.320327:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973176832:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829659.320400:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829659.320413:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829659/real 1787829659] req@ffff9c1a49d8e1c0 x1874668973176832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829659.320427:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973176832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829659.320437:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829659.320441:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829659.320445:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973176832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829659.831436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829659.831495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829660.087604:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829660.855477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829660.855483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829661.111553:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829661.879452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829661.879458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829662.135615:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829662.903442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829662.903479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829663.159605:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829663.927453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829663.927453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829664.183519:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829664.439486:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829664.439495:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829664.439516:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829664.439526:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6114 00000100:00080000:0.0:1787829664.439537:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829664.439544:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829664.439581:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973176960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829664.439607:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829664.439610:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829664.439611:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829664.439614:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6114 00000100:00080000:0.0:1787829664.439617:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829664.439620:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829664.439627:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829664.439639:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973177088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829664.439651:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829664.439655:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829664.439657:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829664.439663:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6114 00000100:00100000:1.0:1787829664.439664:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973176960:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829664.439667:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829664.439674:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829664.439680:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829664.439694:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973177216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829664.439706:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829664.439710:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829664.439712:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829664.439715:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6114 00000100:00080000:0.0:1787829664.439721:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829664.439726:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829664.439732:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829664.439737:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829664.439746:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973177344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829664.439757:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829664.439760:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829664.439762:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829664.439765:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6114 00000100:00080000:0.0:1787829664.439769:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829664.439773:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829664.439778:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829664.439786:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff9c1a78860780 x1874668973176960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829664.439792:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973177472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829664.439816:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829664.439818:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829664.439820:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829664.439824:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6114 00000100:00080000:0.0:1787829664.439830:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829664.439837:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829664.439843:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829664.439850:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973176960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829664.439858:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973177600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829664.439867:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829664.439875:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829664.439877:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829664.439878:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829664.439883:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829664.439888:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973176960:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829664.439889:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6114 00000100:00080000:0.0:1787829664.439898:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829664.439905:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973177088:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829664.439906:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829664.439916:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829664.439933:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973177728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829664.439945:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829664.439963:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff9c1a78862940 x1874668973177088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829664.439979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973177088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829664.439989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829664.439993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829664.439997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973177088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829664.440021:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973177216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829664.440043:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829664.440057:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff9c1a78861a40 x1874668973177216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829664.440070:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973177216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829664.440087:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829664.440090:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829664.440094:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973177216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829664.440101:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973177344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829664.440122:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829664.440134:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff9c1a78860f00 x1874668973177344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829664.440147:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973177344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829664.440157:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829664.440160:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829664.440164:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973177344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829664.440170:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973177472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829664.440209:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829664.440223:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff9c1a788621c0 x1874668973177472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829664.440236:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973177472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829664.440245:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829664.440249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829664.440253:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973177472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829664.440260:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973177600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829664.440281:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829664.440293:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff9c1a78863840 x1874668973177600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829664.440307:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973177600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829664.440315:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829664.440318:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829664.440322:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973177600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829664.440328:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973177728:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829664.440390:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829664.440404:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829664/real 1787829664] req@ffff9c1a78862d00 x1874668973177728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829664.440417:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973177728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829664.440426:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829664.440430:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829664.440433:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973177728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829664.951422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829664.951438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829665.207581:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829665.975482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829665.975482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829666.231582:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829666.999435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829666.999436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829667.255532:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829668.023453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829668.023453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829668.279569:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829669.047420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829669.047420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829669.303508:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829669.559440:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829669.559450:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829669.559456:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829669.559466:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6119 00000100:00080000:3.0:1787829669.559478:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829669.559485:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829669.559526:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973177856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829669.559551:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829669.559555:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829669.559556:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829669.559559:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6119 00000100:00080000:3.0:1787829669.559562:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829669.559564:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829669.559574:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829669.559584:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973177984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829669.559592:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829669.559595:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829669.559597:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829669.559603:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6119 00000100:00100000:1.0:1787829669.559604:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973177856:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829669.559606:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829669.559613:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829669.559619:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829669.559633:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973178112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829669.559645:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829669.559650:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829669.559652:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829669.559656:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6119 00000100:00080000:3.0:1787829669.559661:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829669.559665:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829669.559670:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829669.559681:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829669.559682:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973178240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829669.559692:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829669.559695:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829669.559697:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829669.559701:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6119 00000100:00080000:3.0:1787829669.559704:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829669.559706:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829669.559709:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829669.559719:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973178368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829669.559726:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829669/real 1787829669] req@ffff9c1a7895d2c0 x1874668973177856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829669.559728:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829669.559735:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829669.559737:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829669.559746:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6119 00000100:00080000:3.0:1787829669.559749:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829669.559752:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829669.559756:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829669.559758:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973177856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829669.559767:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973178496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829669.559771:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829669.559776:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829669.559779:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829669.559781:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829669.559785:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829669.559786:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973177856:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829669.559793:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6119 00000100:00080000:3.0:1787829669.559796:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829669.559799:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829669.559801:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829669.559807:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973178624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829669.559823:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973177984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829669.559842:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829669.559850:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829669/real 1787829669] req@ffff9c1a7895cb40 x1874668973177984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829669.559859:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973177984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829669.559865:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829669.559867:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829669.559869:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973177984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829669.559873:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973178112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829669.559883:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829669.559890:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829669/real 1787829669] req@ffff9c1a7895da40 x1874668973178112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829669.559898:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973178112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829669.559903:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829669.559905:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829669.559907:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973178112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829669.559910:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973178240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829669.559920:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829669.559926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829669/real 1787829669] req@ffff9c1a7895e1c0 x1874668973178240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829669.559933:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973178240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829669.559939:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829669.559940:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829669.559942:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973178240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829669.559945:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973178368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829669.559954:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829669.559960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829669/real 1787829669] req@ffff9c1a7895fc00 x1874668973178368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829669.559967:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973178368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829669.559973:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829669.559975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829669.559977:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973178368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829669.559981:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973178496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829669.559991:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829669.559996:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829669/real 1787829669] req@ffff9c1a7895f840 x1874668973178496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829669.560003:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973178496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829669.560009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829669.560011:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829669.560013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973178496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829669.560017:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973178624:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829669.560026:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829669.560031:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829669/real 1787829669] req@ffff9c1a7895f0c0 x1874668973178624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829669.560038:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973178624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829669.560044:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829669.560045:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829669.560048:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973178624:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829670.071487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829670.071487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829670.327635:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829671.095425:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829671.095473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829671.351606:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829672.119494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829672.119501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829672.375656:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829673.143475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829673.143486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829673.399563:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829674.167475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829674.167475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829674.423572:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829674.679542:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829674.679583:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829674.679589:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829674.679605:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6125 00000100:00080000:1.0:1787829674.679636:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829674.679647:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829674.679714:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973178752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829674.679751:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829674.679757:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829674.679760:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829674.679766:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6125 00000100:00080000:1.0:1787829674.679772:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829674.679779:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829674.679808:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829674.679819:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973178752:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829674.679829:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973178880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829674.679843:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829674.679847:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829674.679850:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829674.679855:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6125 00000100:00080000:1.0:1787829674.679859:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829674.679864:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829674.679871:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829674.679883:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973179008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829674.679896:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829674.679900:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829674.679902:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829674.679906:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6125 00000800:00000100:0.0:1787829674.679911:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829674.679924:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829674.679931:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829674.679953:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829674.679967:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973179136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829674.679979:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829674.679981:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829674/real 1787829674] req@ffff9c1a78a7b0c0 x1874668973178752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829674.679985:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829674.679990:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829674.680002:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6125 00000100:00080000:1.0:1787829674.680008:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829674.680013:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829674.680019:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973178752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829674.680022:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829674.680040:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829674.680042:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973179264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829674.680049:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829674.680060:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973178752:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829674.680065:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829674.680071:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829674.680074:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829674.680080:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6125 00000100:00080000:1.0:1787829674.680084:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829674.680093:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973178880:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829674.680093:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829674.680105:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829674.680139:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973179392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829674.680169:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829674.680169:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829674.680177:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829674.680181:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829674.680192:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829674/real 1787829674] req@ffff9c1a78a7a1c0 x1874668973178880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829674.680198:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6125 00000100:00080000:1.0:1787829674.680208:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829674.680216:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973178880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829674.680218:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829674.680245:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829674.680249:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829674.680256:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829674.680262:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973178880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829674.680268:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973179520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829674.680275:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973179008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829674.680311:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829674.680345:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829674/real 1787829674] req@ffff9c1a78a783c0 x1874668973179008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829674.680371:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973179008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829674.680396:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829674.680400:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829674.680404:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973179008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829674.680441:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973179136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829674.680463:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829674.680474:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829674/real 1787829674] req@ffff9c1a78a79e00 x1874668973179136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829674.680485:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973179136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829674.680509:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829674.680513:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829674.680516:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973179136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829674.680523:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973179264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829674.680544:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829674.680571:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829674/real 1787829674] req@ffff9c1a78a7a940 x1874668973179264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829674.680585:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973179264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829674.680602:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829674.680606:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829674.680610:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973179264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829674.680644:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973179392:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829674.680667:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829674.680679:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829674/real 1787829674] req@ffff9c1a78a78780 x1874668973179392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829674.680708:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973179392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829674.680716:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829674.680720:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829674.680723:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973179392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829674.680730:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973179520:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829674.680747:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829674.680774:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829674/real 1787829674] req@ffff9c1a78a7bc00 x1874668973179520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829674.680787:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973179520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829674.680796:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829674.680800:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829674.680805:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973179520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829675.191458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829675.191458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829675.447551:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829676.215478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829676.215478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829676.471560:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829677.239446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829677.239450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829677.495556:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829678.263471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829678.263476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829678.519594:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829679.287432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829679.287476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829679.544014:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829679.799481:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829679.799497:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829679.799503:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829679.799516:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6130 00000100:00080000:1.0:1787829679.799538:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829679.799548:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829679.799599:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973179648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829679.799637:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829679.799642:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829679.799648:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829679.799653:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6130 00000100:00080000:1.0:1787829679.799658:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829679.799663:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829679.799675:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829679.799692:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973179776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829679.799703:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973179648:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829679.799709:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829679.799715:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829679.799718:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829679.799723:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6130 00000100:00080000:1.0:1787829679.799727:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829679.799732:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829679.799738:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829679.799753:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973179904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829679.799766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829679.799771:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829679.799773:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829679.799777:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6130 00000100:00080000:1.0:1787829679.799781:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829679.799784:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829679.799790:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829679.799797:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829679.799811:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973180032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829679.799823:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829679.799827:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829679.799830:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829679.799833:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829679/real 1787829679] req@ffff9c1a78a792c0 x1874668973179648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829679.799838:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6130 00000100:00080000:1.0:1787829679.799846:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829679.799853:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829679.799858:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973179648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829679.799860:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829679.799875:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829679.799880:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973180160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829679.799885:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829679.799895:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973179648:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829679.799901:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829679.799907:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829679.799910:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829679.799915:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6130 00000100:00080000:1.0:1787829679.799920:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829679.799925:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829679.799932:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829679.799933:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973179776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829679.799952:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973180288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829679.799964:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829679.799965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829679.799971:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829679.799974:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829679.799979:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829679/real 1787829679] req@ffff9c1a78a78000 x1874668973179776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829679.799980:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6130 00000100:00080000:1.0:1787829679.799988:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829679.799996:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973179776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829679.799996:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829679.800007:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829679.800009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829679.800015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829679.800019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973179776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829679.800024:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973180416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829679.800027:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973179904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829679.800051:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829679.800060:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829679/real 1787829679] req@ffff9c1a78a79a40 x1874668973179904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829679.800069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973179904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829679.800074:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829679.800077:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829679.800094:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973179904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829679.800100:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973180032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829679.800122:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829679.800135:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829679/real 1787829679] req@ffff9c1a78a7b480 x1874668973180032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829679.800147:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973180032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829679.800154:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829679.800156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829679.800158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973180032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829679.800163:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973180160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829679.800178:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829679.800186:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829679/real 1787829679] req@ffff9c1a78a78f00 x1874668973180160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829679.800193:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973180160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829679.800199:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829679.800201:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829679.800203:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973180160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829679.800216:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973180288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829679.800227:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829679.800232:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829679/real 1787829679] req@ffff9c1a78a7a580 x1874668973180288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829679.800240:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973180288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829679.800246:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829679.800248:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829679.800250:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973180288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829679.800253:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973180416:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829679.800262:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829679.800268:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829679/real 1787829679] req@ffff9c1a78a7b840 x1874668973180416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829679.800275:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973180416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829679.800281:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829679.800283:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829679.800285:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973180416:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829680.311433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829680.311433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829680.567537:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829681.335409:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829681.335414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829681.591536:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829682.359446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829682.359446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829682.615587:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829683.383465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829683.383476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829683.639500:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829684.407431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829684.407431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829684.663537:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829684.919443:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829684.919450:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829684.919454:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829684.919462:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6135 00000100:00080000:1.0:1787829684.919473:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829684.919479:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829684.919514:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973180544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829684.919542:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829684.919545:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829684.919546:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829684.919549:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6135 00000100:00080000:1.0:1787829684.919552:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829684.919554:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829684.919561:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829684.919570:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973180672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829684.919577:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829684.919580:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829684.919581:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829684.919584:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6135 00000100:00080000:1.0:1787829684.919585:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829684.919588:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829684.919590:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829684.919597:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973180800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829684.919605:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973180544:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829684.919608:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829684.919614:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829684.919617:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829684.919628:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6135 00000100:00080000:1.0:1787829684.919631:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829684.919634:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829684.919637:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829684.919645:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973180928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829684.919660:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829684.919662:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829684.919664:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829684.919667:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6135 00000100:00080000:1.0:1787829684.919677:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829684.919680:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829684.919683:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829684.919694:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973181056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829684.919696:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829684.919706:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829684.919709:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829684.919710:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829684.919713:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6135 00000100:00080000:1.0:1787829684.919716:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829684.919718:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829684.919722:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829684.919730:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973181184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829684.919751:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829684.919754:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829684.919755:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829684.919762:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829684/real 1787829684] req@ffff9c1a78a7b840 x1874668973180544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829684.919764:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6135 00000100:00080000:1.0:1787829684.919768:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829684.919774:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829684.919780:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829684.919792:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973181312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829684.919801:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973180544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829684.919815:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829684.919821:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829684.919827:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973180544:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829684.919837:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973180672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829684.919867:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829684.919882:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829684/real 1787829684] req@ffff9c1a78a7a580 x1874668973180672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829684.919896:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973180672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829684.919906:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829684.919910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829684.919913:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973180672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829684.919937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973180800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829684.919968:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829684.919980:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829684/real 1787829684] req@ffff9c1a78a78f00 x1874668973180800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829684.919995:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973180800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829684.920005:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829684.920009:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829684.920013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973180800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829684.920019:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973180928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829684.920040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829684.920069:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829684/real 1787829684] req@ffff9c1a78a7b480 x1874668973180928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829684.920087:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973180928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829684.920108:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829684.920111:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829684.920116:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973180928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829684.920122:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973181056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829684.920148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829684.920161:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829684/real 1787829684] req@ffff9c1a78a79a40 x1874668973181056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829684.920174:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973181056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829684.920183:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829684.920186:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829684.920190:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973181056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829684.920197:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973181184:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829684.920218:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829684.920230:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829684/real 1787829684] req@ffff9c1a78a78000 x1874668973181184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829684.920243:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973181184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829684.920252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829684.920255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829684.920259:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973181184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829684.920266:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973181312:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829684.920289:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829684.920302:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829684/real 1787829684] req@ffff9c1a78a792c0 x1874668973181312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829684.920315:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973181312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829684.920324:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829684.920327:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829684.920330:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973181312:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829685.431463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829685.431471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829685.687552:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829686.455423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829686.455431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829686.711548:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829687.479479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829687.479479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829687.735498:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829688.503403:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829688.503422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829688.759493:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829689.527449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829689.527449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829689.783593:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829690.039417:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829690.039428:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829690.039432:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.039443:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6140 00000100:00080000:1.0:1787829690.039454:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.039460:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829690.039496:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973181440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829690.039534:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829690.039537:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829690.039539:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.039542:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6140 00000100:00080000:1.0:1787829690.039546:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.039549:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829690.039557:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829690.039569:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973181568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829690.039578:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829690.039582:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973181440:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829690.039582:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829690.039585:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.039592:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6140 00000100:00080000:1.0:1787829690.039594:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.039596:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829690.039599:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829690.039608:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973181696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829690.039616:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829690.039618:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829690.039620:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.039623:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6140 00000100:00080000:1.0:1787829690.039632:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.039635:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829690.039637:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829690.039643:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829690.039647:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973181824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829690.039654:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829690.039657:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829690.039658:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.039661:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6140 00000100:00080000:1.0:1787829690.039663:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.039665:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829690.039668:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829690.039677:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973181952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829690.039683:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff9c1a78a7bc00 x1874668973181440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829690.039687:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829690.039696:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829690.039700:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829690.039705:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973181440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829690.039707:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6140 00000100:00080000:1.0:1787829690.039714:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829690.039719:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.039724:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829690.039726:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973181440:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829690.039730:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829690.039737:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829690.039748:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973181568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829690.039750:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973182080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829690.039760:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829690.039763:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829690.039764:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829690.039769:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6140 00000800:00000100:0.0:1787829690.039774:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829690.039776:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829690.039781:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829690.039786:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829690.039790:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff9c1a78a78780 x1874668973181568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829690.039805:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973181568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829690.039805:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973182208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829690.039835:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.039839:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829690.039842:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973181568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829690.039847:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973181696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829690.039860:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829690.039867:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff9c1a78a7a940 x1874668973181696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829690.039875:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973181696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829690.039880:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.039882:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829690.039885:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973181696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829690.039888:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973181824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829690.039897:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829690.039902:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff9c1a78a79e00 x1874668973181824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829690.039910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973181824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829690.039915:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.039917:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829690.039920:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973181824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829690.039923:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973181952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829690.039932:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829690.039937:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff9c1a78a783c0 x1874668973181952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829690.039944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973181952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829690.039950:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.039951:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829690.039953:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973181952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829690.039963:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973182080:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829690.039973:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829690.039979:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff9c1a78a7a1c0 x1874668973182080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829690.039986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973182080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829690.039991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.039993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829690.039995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973182080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829690.039999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973182208:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829690.040020:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829690.040031:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829690/real 1787829690] req@ffff9c1a78a7b0c0 x1874668973182208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829690.040039:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973182208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829690.040044:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829690.040046:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829690.040049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973182208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829690.551432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829690.551432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829690.807512:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829691.575418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829691.575428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829691.831545:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829692.599426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829692.599426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829692.855552:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829693.623481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829693.623489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829693.879570:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829694.647446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829694.647473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829694.903528:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829695.159419:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829695.159426:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829695.159430:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829695.159439:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6145 00000100:00080000:1.0:1787829695.159448:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829695.159454:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829695.159487:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973182336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829695.159509:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829695.159512:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829695.159513:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829695.159516:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6145 00000100:00080000:1.0:1787829695.159518:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829695.159521:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829695.159528:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829695.159537:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973182464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829695.159544:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829695.159546:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829695.159548:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829695.159557:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6145 00000100:00100000:0.0:1787829695.159561:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973182336:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829695.159561:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829695.159568:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829695.159573:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829695.159581:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973182592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829695.159588:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829695.159591:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829695.159593:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829695.159595:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6145 00000100:00080000:1.0:1787829695.159608:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829695.159611:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829695.159613:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829695.159620:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829695.159624:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973182720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829695.159632:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829695.159634:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829695.159636:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829695.159639:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6145 00000100:00080000:1.0:1787829695.159642:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829695.159644:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829695.159646:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829695.159655:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973182848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829695.159661:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff9c1a78a7b0c0 x1874668973182336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829695.159664:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829695.159672:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829695.159675:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829695.159680:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6145 00000100:00080000:1.0:1787829695.159683:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829695.159686:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973182336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829695.159695:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787829695.159698:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829695.159699:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829695.159704:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829695.159709:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973182336:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829695.159712:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973182976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829695.159718:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973182464:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829695.159726:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829695.159729:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829695.159730:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829695.159733:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6145 00000100:00080000:1.0:1787829695.159738:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829695.159743:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829695.159744:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829695.159751:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829695.159758:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff9c1a78a7a1c0 x1874668973182464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829695.159763:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973183104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829695.159777:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973182464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.159784:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.159787:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829695.159789:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973182464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829695.159808:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973182592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829695.159819:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829695.159825:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff9c1a78a783c0 x1874668973182592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829695.159832:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973182592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.159837:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.159839:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829695.159841:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973182592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829695.159845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973182720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829695.159854:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829695.159860:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff9c1a78a79e00 x1874668973182720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829695.159867:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973182720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.159872:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.159874:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829695.159876:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973182720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829695.159879:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973182848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829695.159889:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829695.159895:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff9c1a78a7a940 x1874668973182848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829695.159902:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973182848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.159907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.159908:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829695.159910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973182848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829695.159913:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973182976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829695.159923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829695.159929:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff9c1a78a78780 x1874668973182976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829695.159936:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973182976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.159941:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.159943:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829695.159945:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973182976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829695.159948:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973183104:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829695.159957:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829695.159962:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829695/real 1787829695] req@ffff9c1a78a7bc00 x1874668973183104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829695.159969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973183104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829695.159989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829695.159991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829695.159993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973183104:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829695.671446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829695.671454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829695.927519:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829696.695415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829696.695415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829696.951552:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829697.719439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829697.719439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829697.975528:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829698.743432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829698.743432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829698.999482:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829699.767482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829699.767482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829700.023575:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829700.279474:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829700.279491:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829700.279496:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829700.279509:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6150 00000100:00080000:3.0:1787829700.279520:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829700.279527:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829700.279563:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973183232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829700.279590:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829700.279594:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829700.279596:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829700.279600:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6150 00000100:00080000:3.0:1787829700.279604:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829700.279608:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829700.279618:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829700.279635:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973183360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829700.279640:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973183232:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829700.279657:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829700.279661:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829700.279664:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829700.279670:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6150 00000100:00080000:3.0:1787829700.279674:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829700.279679:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829700.279685:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829700.279699:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973183488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829700.279711:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829700.279714:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829700.279717:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829700.279721:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6150 00000100:00080000:3.0:1787829700.279725:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829700.279730:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829700.279737:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829700.279738:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973183360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829700.279755:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973183616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829700.279768:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829700.279772:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829700.279775:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829700.279780:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6150 00000100:00100000:0.0:1787829700.279784:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973183488:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829700.279786:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829700.279795:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829700.279802:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829700.279815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973183744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829700.279826:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829700.279827:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973183616:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829700.279835:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829700.279839:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829700.279844:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6150 00000100:00080000:3.0:1787829700.279849:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829700.279853:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829700.279860:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829700.279865:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973183744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829700.279874:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973183872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829700.279887:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829700.279890:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829700.279893:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829700.279895:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973183872:192.168.203.148@tcp:8:kworker.0 00000100:00080000:3.0:1787829700.279900:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6150 00000100:00080000:3.0:1787829700.279907:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829700.279911:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829700.279917:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829700.279930:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973184000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829700.279976:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973184000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829700.791413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829700.791438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829701.047632:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829701.815425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829701.815427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829702.071535:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829702.839437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829702.839466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829703.095468:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829703.351817:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787829703.352098:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829703.352128:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829703.352231:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829703.352249:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829703.352270:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829703.352283:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829703.352298:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829703.352662:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787829703.352670:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829703] req@ffff9c1a7895da40 x1874668973183872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.352720:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973183872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.352738:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.352747:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829703.352754:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973183872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829703.352767:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829703] req@ffff9c1a7895e1c0 x1874668973183744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.352792:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973183744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.352806:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.352812:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829703.352817:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973183744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829703.352829:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829703] req@ffff9c1a7895fc00 x1874668973183616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.352861:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973183616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.352875:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.352881:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829703.352886:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973183616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829703.352897:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829703] req@ffff9c1a7895f0c0 x1874668973183360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.352942:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973183360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.352955:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.352960:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829703.352965:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973183360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829703.352975:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829703] req@ffff9c1a7895f840 x1874668973183488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.352994:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973183488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.353009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.353015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829703.353022:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973183488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829703.353032:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829703] req@ffff9c1a7895de00 x1874668973183232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.353052:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973183232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.353065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.353070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829703.353074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973183232:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829703.353131:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829700/real 1787829703] req@ffff9c1a7895cb40 x1874668973184000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.353152:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973184000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.353164:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.353168:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829703.353174:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973184000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829703.863440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829703.863440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829704.119501:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829704.887450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829704.887489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829705.143503:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829705.399423:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829705.399432:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829705.399435:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829705.399444:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6155 00000100:00080000:3.0:1787829705.399457:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829705.399463:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829705.399492:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973184128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829705.399517:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829705.399520:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829705.399521:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829705.399524:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6155 00000100:00080000:3.0:1787829705.399526:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829705.399529:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829705.399537:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829705.399547:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973184256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829705.399555:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829705.399558:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829705.399559:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973184128:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829705.399559:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829705.399567:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6155 00000100:00080000:3.0:1787829705.399576:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829705.399578:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829705.399582:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829705.399592:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973184384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829705.399600:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829705.399603:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829705.399604:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829705.399607:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6155 00000100:00080000:3.0:1787829705.399610:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829705.399612:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829705.399615:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829705.399622:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973184512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829705.399629:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829705.399634:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829705.399635:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829705.399643:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829705.399650:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6155 00000100:00080000:3.0:1787829705.399663:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829705.399667:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829705.399673:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829705.399683:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829705] req@ffff9c1a7895d2c0 x1874668973184128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829705.399688:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973184640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829705.399715:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973184128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829705.399722:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829705.399732:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829705.399736:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829705.399746:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6155 00000100:00080000:1.0:1787829705.399750:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829705.399752:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829705.399761:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829705.399762:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829705.399769:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973184128:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787829705.399771:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829705.399788:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973184768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829705.399799:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973184256:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829705.399807:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829705.399812:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829705.399814:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829705.399819:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6155 00000100:00080000:3.0:1787829705.399825:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829705.399830:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829705.399832:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829705.399840:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829705.399850:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829705] req@ffff9c1a7895f480 x1874668973184256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829705.399855:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973184896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829705.399879:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973184256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829705.399889:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829705.399894:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829705.399898:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973184256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829705.399917:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973184384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829705.399939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829705.399951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829705] req@ffff9c1a7895c000 x1874668973184384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829705.399963:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973184384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829705.399972:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829705.399975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829705.399979:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973184384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829705.399985:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973184512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829705.400004:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829705.400015:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829705] req@ffff9c1a7895e580 x1874668973184512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829705.400026:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973184512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829705.400035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829705.400038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829705.400042:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973184512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829705.400048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973184640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829705.400066:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829705.400077:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829705] req@ffff9c1a7895cf00 x1874668973184640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829705.400089:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973184640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829705.400097:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829705.400100:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829705.400103:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973184640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829705.400128:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973184768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829705.400149:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829705.400161:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829705] req@ffff9c1a7895e940 x1874668973184768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829705.400181:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973184768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829705.400189:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829705.400192:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829705.400195:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973184768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829705.400200:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973184896:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829705.400221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829705.400232:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829705/real 1787829705] req@ffff9c1a7895c780 x1874668973184896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829705.400245:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973184896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829705.400253:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829705.400257:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829705.400260:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973184896:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829705.911424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829705.911475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829706.167529:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829706.935474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829706.935474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829707.191529:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829707.959424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829707.959424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829708.215487:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829708.983453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829708.983453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829709.239545:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829710.007430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829710.007430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829710.263511:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829710.519517:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829710.519527:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829710.519531:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829710.519540:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6160 00000100:00080000:3.0:1787829710.519549:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829710.519555:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829710.519583:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973185024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829710.519605:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829710.519608:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829710.519610:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829710.519614:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6160 00000100:00080000:3.0:1787829710.519617:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829710.519620:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829710.519628:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829710.519640:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973185152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829710.519651:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829710.519653:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973185024:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829710.519662:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829710.519664:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829710.519668:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6160 00000100:00080000:3.0:1787829710.519671:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829710.519673:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829710.519677:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829710.519686:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973185280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829710.519694:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829710.519696:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829710.519698:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829710.519700:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6160 00000100:00080000:3.0:1787829710.519709:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829710.519711:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829710.519715:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829710.519721:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829710.519725:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973185408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829710.519737:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829710.519741:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829710.519742:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829710.519747:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6160 00000100:00100000:2.0:1787829710.519757:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829710/real 1787829710] req@ffff9c1a7895c3c0 x1874668973185024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829710.519772:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973185024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829710.519780:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829710.519783:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829710.519786:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973185024:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829710.519803:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973185152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829710.519821:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829710.519832:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829710/real 1787829710] req@ffff9c1a7895c780 x1874668973185152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829710.519833:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829710.519843:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829710.519846:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973185152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829710.519851:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829710.519855:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829710.519859:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829710.519862:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973185152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829710.519882:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973185536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829710.519893:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829710.519897:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973185280:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829710.519897:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829710.519900:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829710.519906:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6160 00000100:00080000:3.0:1787829710.519910:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829710.519913:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829710.519918:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829710.519928:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973185664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829710.519934:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829710.519940:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829710.519955:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829710/real 1787829710] req@ffff9c1a7895e940 x1874668973185280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829710.519970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973185280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829710.519980:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829710.519982:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829710.519984:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973185280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829710.519990:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973185408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829710.520009:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829710.520019:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829710/real 1787829710] req@ffff9c1a7895cf00 x1874668973185408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829710.520028:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973185408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829710.520036:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829710.520040:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829710.520041:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829710.520048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829710.520050:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6160 00000100:00100000:2.0:1787829710.520055:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973185408:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829710.520057:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829710.520064:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829710.520068:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829710.520092:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973185536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829710.520111:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829710.520125:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973185792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829710.520129:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829710/real 1787829710] req@ffff9c1a7895e580 x1874668973185536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829710.520162:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973185536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829710.520168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829710.520172:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829710.520175:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973185536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829710.520183:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973185664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829710.520207:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829710.520224:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829710/real 1787829710] req@ffff9c1a7895c000 x1874668973185664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829710.520239:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973185664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829710.520246:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829710.520249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829710.520253:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973185664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829710.520269:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973185792:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829710.520285:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829710.520293:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829710/real 1787829710] req@ffff9c1a7895f480 x1874668973185792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829710.520301:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973185792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829710.520306:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829710.520308:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829710.520310:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973185792:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829711.031424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829711.031427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829711.287521:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829712.055440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829712.055440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829712.311505:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829713.079457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829713.079457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829713.335523:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829714.103433:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829714.103458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829714.359495:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829715.127452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829715.127452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829715.383513:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829715.639464:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829715.639473:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829715.639478:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829715.639487:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6165 00000100:00080000:3.0:1787829715.639498:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829715.639504:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829715.639539:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973185920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829715.639576:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829715.639580:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829715.639582:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829715.639586:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6165 00000100:00080000:3.0:1787829715.639590:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829715.639594:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829715.639604:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829715.639618:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973186048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829715.639635:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973185920:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829715.639640:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829715.639645:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829715.639647:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829715.639652:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6165 00000100:00080000:3.0:1787829715.639656:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829715.639659:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829715.639664:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829715.639675:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973186176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829715.639686:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829715.639690:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829715.639692:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829715.639697:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6165 00000800:00000100:2.0:1787829715.639699:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829715.639702:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829715.639708:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829715.639713:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829715.639724:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973186304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829715.639733:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829715.639737:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829715.639741:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829715/real 1787829715] req@ffff9c1a7895d2c0 x1874668973185920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829715.639741:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829715.639748:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6165 00000100:00080000:3.0:1787829715.639754:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829715.639760:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829715.639766:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829715.639771:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973185920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829715.639778:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973186432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829715.639792:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829715.639795:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829715.639800:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829715.639801:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829715.639806:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829715.639809:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973185920:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829715.639813:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6165 00000100:00080000:3.0:1787829715.639819:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829715.639823:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829715.639828:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829715.639853:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973186560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829715.639854:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973186048:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829715.639872:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829715.639876:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829715.639878:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829715.639883:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6165 00000100:00080000:3.0:1787829715.639887:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829715.639890:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829715.639894:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829715.639901:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829715.639910:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829715/real 1787829715] req@ffff9c1a7895cb40 x1874668973186048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829715.639915:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973186688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829715.639941:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973186048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829715.639956:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829715.639960:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829715.639965:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973186048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829715.639973:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973186176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829715.640007:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829715.640018:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829715/real 1787829715] req@ffff9c1a7895de00 x1874668973186176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829715.640031:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973186176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829715.640039:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829715.640042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829715.640046:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973186176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829715.640052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973186304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829715.640069:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829715.640078:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829715/real 1787829715] req@ffff9c1a7895f840 x1874668973186304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829715.640089:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973186304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829715.640099:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829715.640101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829715.640105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973186304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829715.640110:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973186432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829715.640126:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829715.640135:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829715/real 1787829715] req@ffff9c1a7895f0c0 x1874668973186432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829715.640147:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973186432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829715.640155:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829715.640158:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829715.640162:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973186432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829715.640180:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973186560:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829715.640197:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829715.640207:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829715/real 1787829715] req@ffff9c1a7895fc00 x1874668973186560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829715.640219:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973186560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829715.640227:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829715.640230:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829715.640234:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973186560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829715.640240:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973186688:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829715.640255:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829715.640264:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829715/real 1787829715] req@ffff9c1a7895e1c0 x1874668973186688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829715.640276:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973186688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829715.640284:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829715.640287:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829715.640291:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973186688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829716.151418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829716.151425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829716.407504:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829717.175417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829717.175417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829717.431537:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829718.199421:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829718.199423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829718.455523:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829719.223440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829719.223441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829719.479545:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829720.247461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829720.247572:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829720.503504:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829720.759391:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829720.759400:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829720.759403:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829720.759412:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6171 00000100:00080000:3.0:1787829720.759423:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829720.759431:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829720.759468:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973186816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829720.759489:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829720.759495:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829720.759497:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829720.759501:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6171 00000100:00080000:3.0:1787829720.759506:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829720.759511:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829720.759521:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829720.759529:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973186816:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829720.759538:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973186944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829720.759549:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829720.759551:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829720.759552:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829720.759555:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6171 00000100:00080000:3.0:1787829720.759559:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829720.759563:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829720.759567:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829720.759577:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973187072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829720.759585:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829720.759588:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829720.759590:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829720.759593:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829720.759596:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6171 00000100:00080000:3.0:1787829720.759600:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829720.759602:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829720.759606:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829720.759614:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973187200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829720.759621:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829720.759625:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829720/real 1787829720] req@ffff9c1a7895da40 x1874668973186816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829720.759626:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829720.759629:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829720.759636:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6171 00000100:00080000:3.0:1787829720.759638:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829720.759640:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829720.759642:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829720.759650:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973187328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829720.759661:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973186816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829720.759664:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829720.759672:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829720.759675:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829720.759682:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6171 00000100:00080000:2.0:1787829720.759684:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829720.759685:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829720.759692:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829720.759693:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787829720.759698:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829720.759699:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973186816:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829720.759708:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973187456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829720.759714:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829720.759716:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829720.759718:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829720.759723:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6171 00000100:00100000:2.0:1787829720.759724:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973186944:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829720.759727:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829720.759730:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829720.759733:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829720.759742:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973187584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829720.759750:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829720.759778:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829720/real 1787829720] req@ffff9c1a7895e1c0 x1874668973186944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829720.759817:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973186944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829720.759827:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829720.759830:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829720.759833:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973186944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829720.759838:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973187072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829720.759864:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829720.759874:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829720/real 1787829720] req@ffff9c1a7895fc00 x1874668973187072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829720.759882:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973187072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829720.759898:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829720.759901:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829720.759905:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973187072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829720.759912:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973187200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829720.759934:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829720.759946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829720/real 1787829720] req@ffff9c1a7895f0c0 x1874668973187200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829720.759954:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973187200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829720.759971:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829720.759973:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829720.759975:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973187200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829720.759979:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973187328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829720.759992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829720.759998:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829720/real 1787829720] req@ffff9c1a7895f840 x1874668973187328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829720.760005:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973187328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829720.760011:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829720.760012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829720.760014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973187328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829720.760018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973187456:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829720.760040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829720.760046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829720/real 1787829720] req@ffff9c1a7895de00 x1874668973187456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829720.760054:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973187456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829720.760059:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829720.760060:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829720.760063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973187456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829720.760077:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973187584:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829720.760088:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829720.760094:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829720/real 1787829720] req@ffff9c1a7895cb40 x1874668973187584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829720.760101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973187584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829720.760107:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829720.760109:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829720.760111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973187584:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829721.271445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829721.271446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829721.527568:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829722.295446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829722.295446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829722.551478:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829723.319420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829723.319428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829723.575523:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829724.343517:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829724.343517:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829724.599539:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829725.367442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829725.367444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829725.623502:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829725.879449:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829725.879459:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829725.879464:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829725.879474:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6176 00000100:00080000:3.0:1787829725.879486:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829725.879493:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829725.879538:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973187712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829725.879564:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829725.879568:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829725.879569:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829725.879574:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6176 00000100:00080000:3.0:1787829725.879578:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829725.879582:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829725.879591:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829725.879606:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973187840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829725.879615:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829725.879619:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829725.879620:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829725.879624:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6176 00000100:00080000:3.0:1787829725.879628:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829725.879633:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829725.879638:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829725.879651:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973187712:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829725.879652:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973187968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829725.879679:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829725.879682:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829725.879683:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829725.879687:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6176 00000100:00080000:3.0:1787829725.879690:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829725.879694:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829725.879698:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829725.879707:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973188096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829725.879714:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829725.879717:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829725.879719:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829725.879722:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6176 00000100:00080000:3.0:1787829725.879744:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829725.879747:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829725.879750:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829725.879771:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973188224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829725.879777:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829725.879788:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829725.879790:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829725.879792:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829725.879796:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6176 00000100:00080000:3.0:1787829725.879800:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829725.879803:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829725.879808:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829725.879816:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973188352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829725.879822:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829725.879825:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829725.879826:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829725.879831:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6176 00000100:00080000:3.0:1787829725.879834:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829725.879839:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829725.879842:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829725.879852:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829725/real 1787829725] req@ffff9c1a7895d2c0 x1874668973187712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829725.879853:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973188480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829725.879913:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973187712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829725.879928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829725.879933:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829725.879937:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973187712:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829725.879947:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973187840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829725.879976:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829725.879992:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829725/real 1787829725] req@ffff9c1a7895f480 x1874668973187840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829725.880006:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973187840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829725.880015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829725.880019:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829725.880023:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973187840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829725.880058:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973187968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829725.880080:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829725.880093:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829725/real 1787829725] req@ffff9c1a7895c000 x1874668973187968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829725.880119:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973187968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829725.880127:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829725.880131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829725.880135:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973187968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829725.880142:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973188096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829725.880172:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829725.880185:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829725/real 1787829725] req@ffff9c1a7895e580 x1874668973188096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829725.880198:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973188096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829725.880208:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829725.880211:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829725.880215:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973188096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829725.880222:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973188224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829725.880244:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829725.880256:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829725/real 1787829725] req@ffff9c1a7895cf00 x1874668973188224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829725.880268:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973188224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829725.880277:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829725.880280:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829725.880284:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973188224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829725.880291:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973188352:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829725.880312:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829725.880324:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829725/real 1787829725] req@ffff9c1a7895e940 x1874668973188352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829725.880336:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973188352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829725.880344:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829725.880383:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829725.880388:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973188352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829725.880395:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973188480:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829725.880417:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829725.880429:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829725/real 1787829725] req@ffff9c1a7895c780 x1874668973188480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829725.880442:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973188480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829725.880451:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829725.880454:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829725.880457:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973188480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829726.391494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829726.391516:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829726.647512:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829727.415457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829727.415489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829727.671539:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829728.439467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829728.439482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829728.695472:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829729.463399:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829729.463415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829729.719470:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829730.487477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829730.487520:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829730.743548:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829730.999435:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829730.999453:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829730.999458:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829730.999469:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6181 00000100:00080000:3.0:1787829730.999481:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829730.999488:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829730.999538:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973188608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829730.999573:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829730.999576:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829730.999579:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829730.999583:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6181 00000100:00080000:3.0:1787829730.999586:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829730.999589:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829730.999601:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829730.999615:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973188736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829730.999622:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973188608:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829730.999627:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829730.999631:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829730.999632:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829730.999636:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6181 00000100:00080000:3.0:1787829730.999638:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829730.999640:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829730.999644:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829730.999653:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973188864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829730.999662:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829730.999665:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829730.999666:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829730.999669:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6181 00000100:00080000:3.0:1787829730.999686:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829730.999689:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829730.999692:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829730.999698:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829730.999703:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973188992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829730.999711:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829730.999714:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829730.999716:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829730.999719:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6181 00000100:00080000:3.0:1787829730.999737:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829730.999742:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829730.999749:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829730.999761:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829730/real 1787829730] req@ffff9c1a7895c3c0 x1874668973188608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829730.999765:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973189120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829730.999787:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973188608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829730.999794:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829730.999799:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829730.999801:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829730.999802:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829730.999808:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829730.999814:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6181 00000100:00100000:2.0:1787829730.999815:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973188608:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829730.999823:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829730.999831:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829730.999838:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829730.999844:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973188736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829730.999856:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973189248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829730.999867:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829730.999869:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829730.999873:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829730.999876:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829730.999881:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6181 00000100:00080000:3.0:1787829730.999885:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829730.999886:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829730/real 1787829730] req@ffff9c1a7895c780 x1874668973188736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829730.999893:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829730.999901:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829730.999906:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973188736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829730.999914:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829730.999919:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973189376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829730.999920:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829730.999925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973188736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829730.999935:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973188864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829730.999956:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829730.999964:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829730/real 1787829730] req@ffff9c1a7895e940 x1874668973188864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829730.999972:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973188864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829730.999978:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829730.999979:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829730.999981:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973188864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829730.999985:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973188992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829730.999996:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829731.000002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829730/real 1787829730] req@ffff9c1a7895cf00 x1874668973188992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829731.000010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973188992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829731.000015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829731.000017:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829731.000019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973188992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829731.000022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973189120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829731.000032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829731.000038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829730/real 1787829731] req@ffff9c1a7895e580 x1874668973189120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829731.000045:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973189120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829731.000050:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829731.000052:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829731.000054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973189120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829731.000067:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973189248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829731.000078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829731.000084:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829730/real 1787829731] req@ffff9c1a7895c000 x1874668973189248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829731.000092:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973189248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829731.000098:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829731.000100:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829731.000102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973189248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829731.000106:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973189376:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829731.000115:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829731.000120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829730/real 1787829731] req@ffff9c1a7895f480 x1874668973189376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829731.000127:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973189376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829731.000133:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829731.000135:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829731.000137:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973189376:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829731.511455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829731.511456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829731.767485:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829732.535463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829732.535463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829732.791605:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829733.559442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829733.559442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829733.815458:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829734.583461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829734.583495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829734.839554:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829735.607468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829735.607489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829735.863546:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829736.119420:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829736.119430:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829736.119434:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829736.119444:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6186 00000100:00080000:3.0:1787829736.119456:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829736.119463:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829736.119499:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973189504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829736.119526:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829736.119530:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829736.119532:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829736.119536:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6186 00000100:00080000:3.0:1787829736.119540:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829736.119545:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829736.119554:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829736.119568:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973189632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829736.119570:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973189504:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829736.119588:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829736.119591:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829736.119594:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829736.119598:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6186 00000100:00080000:3.0:1787829736.119602:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829736.119606:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829736.119611:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829736.119622:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973189760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829736.119631:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829736.119640:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829736.119645:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829736.119647:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829736.119652:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6186 00000100:00080000:3.0:1787829736.119657:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829736.119662:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829736.119667:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff9c1a7895d2c0 x1874668973189504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829736.119669:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829736.119685:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973189888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829736.119700:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973189504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829736.119724:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829736.119733:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829736.119736:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829736.119737:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829736.119743:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829736.119743:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6186 00000100:00100000:2.0:1787829736.119749:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973189504:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829736.119751:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829736.119758:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829736.119763:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829736.119774:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973189632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829736.119777:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973190016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829736.119791:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829736.119794:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787829736.119798:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829736.119798:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829736.119807:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6186 00000100:00080000:3.0:1787829736.119811:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829736.119814:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff9c1a7895cb40 x1874668973189632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829736.119819:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829736.119830:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829736.119833:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973189632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829736.119841:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829736.119846:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829736.119848:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973190144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829736.119850:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973189632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829736.119860:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973189760:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829736.119868:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829736.119873:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829736.119875:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829736.119880:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829736.119884:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6186 00000100:00080000:3.0:1787829736.119890:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829736.119892:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff9c1a7895de00 x1874668973189760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829736.119898:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829736.119907:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973189760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829736.119907:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829736.119917:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829736.119922:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829736.119924:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973190272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829736.119927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973189760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829736.119938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973189888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829736.119955:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829736.119964:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff9c1a7895f840 x1874668973189888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829736.119973:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973189888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829736.119979:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829736.119981:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829736.119983:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973189888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829736.119994:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973190016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829736.120004:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829736.120010:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff9c1a7895f0c0 x1874668973190016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829736.120018:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973190016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829736.120023:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829736.120025:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829736.120027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973190016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829736.120030:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973190144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829736.120040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829736.120045:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff9c1a7895fc00 x1874668973190144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829736.120053:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973190144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829736.120058:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829736.120059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829736.120061:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973190144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829736.120064:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973190272:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829736.120073:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829736.120079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829736/real 1787829736] req@ffff9c1a7895e1c0 x1874668973190272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829736.120094:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973190272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829736.120099:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829736.120101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829736.120103:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973190272:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829736.631426:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829736.631427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829736.887559:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829737.655441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829737.655447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829737.911484:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829738.679449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829738.679449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829738.935519:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829739.703411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829739.703414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829739.959500:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829740.727411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829740.727429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829740.983503:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829741.239452:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.239459:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829741.239463:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.239477:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6191 00000100:00080000:3.0:1787829741.239486:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.239492:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829741.239523:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973190400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829741.239545:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.239548:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829741.239549:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.239552:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6191 00000100:00080000:3.0:1787829741.239554:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.239557:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829741.239564:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829741.239575:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973190528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829741.239583:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.239585:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829741.239586:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.239589:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6191 00000100:00080000:3.0:1787829741.239591:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.239595:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829741.239600:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973190400:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787829741.239600:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829741.239609:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973190656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829741.239617:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.239620:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829741.239621:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.239624:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6191 00000100:00080000:3.0:1787829741.239626:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.239629:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829741.239631:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829741.239640:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973190784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829741.239646:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.239648:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829741.239649:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.239667:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6191 00000100:00080000:3.0:1787829741.239671:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.239674:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829741.239678:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829741.239683:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829741.239691:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973190912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829741.239699:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829741.239702:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829741.239704:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.239707:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6191 00000100:00080000:3.0:1787829741.239709:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.239711:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829741.239715:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829741.239723:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973191040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829741.239731:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829741.239732:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff9c1a7895da40 x1874668973190400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829741.239733:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829741.239736:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829741.239744:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6191 00000100:00080000:3.0:1787829741.239749:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829741.239755:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829741.239759:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829741.239768:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973190400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829741.239773:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973191168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829741.239807:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.239813:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829741.239818:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973190400:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829741.239828:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973190528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829741.239853:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829741.239867:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff9c1a7895e1c0 x1874668973190528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.239881:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973190528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.239890:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.239893:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829741.239897:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973190528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829741.239921:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973190656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829741.239943:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829741.239955:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff9c1a7895fc00 x1874668973190656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.239968:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973190656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.239977:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.239980:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829741.239984:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973190656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829741.239991:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973190784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829741.240011:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829741.240023:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff9c1a7895f0c0 x1874668973190784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.240036:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973190784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.240044:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.240047:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829741.240051:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973190784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829741.240058:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973190912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829741.240078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829741.240089:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff9c1a7895f840 x1874668973190912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.240102:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973190912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.240111:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.240114:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829741.240118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973190912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829741.240124:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973191040:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829741.240143:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829741.240154:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff9c1a7895de00 x1874668973191040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.240168:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973191040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.240177:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.240180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829741.240183:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973191040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829741.240189:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973191168:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829741.240208:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829741.240218:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829741/real 1787829741] req@ffff9c1a7895cb40 x1874668973191168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829741.240231:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973191168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829741.240240:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829741.240244:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829741.240248:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973191168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829741.751405:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829741.751438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829742.007553:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829742.775442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829742.775446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829743.031633:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829743.799479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829743.799482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829744.055482:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829744.823441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829744.823444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829745.079504:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829745.847454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829745.847454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829746.103506:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829746.359463:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829746.359476:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829746.359482:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829746.359494:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6196 00000100:00080000:3.0:1787829746.359508:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829746.359518:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829746.359554:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973191296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829746.359581:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829746.359586:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829746.359588:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829746.359592:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6196 00000100:00080000:3.0:1787829746.359596:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829746.359600:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829746.359610:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829746.359637:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973191296:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829746.359653:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973191424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829746.359668:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829746.359671:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829746.359673:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829746.359678:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6196 00000100:00080000:3.0:1787829746.359683:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829746.359688:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829746.359694:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829746.359706:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973191552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829746.359716:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829746.359724:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829746.359729:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829746.359732:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829746.359738:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6196 00000100:00080000:3.0:1787829746.359741:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829746.359746:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829746.359766:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff9c1a7895d2c0 x1874668973191296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829746.359787:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973191296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829746.359805:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.359809:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829746.359814:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973191296:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787829746.359839:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829746.359844:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973191424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829746.359872:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829746.359892:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff9c1a7895f480 x1874668973191424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829746.359910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973191424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829746.359918:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.359922:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829746.359927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973191424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829746.359936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973191552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829746.359964:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829746.359986:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff9c1a7895c000 x1874668973191552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829746.359996:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973191680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829746.360004:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973191552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829746.360020:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.360024:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829746.360029:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973191552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829746.360043:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973191680:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829746.360063:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829746.360065:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829746.360072:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829746.360075:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829746.360082:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6196 00000100:00080000:3.0:1787829746.360089:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829746.360094:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff9c1a7895e580 x1874668973191680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829746.360098:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829746.360109:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829746.360117:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973191680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829746.360127:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973191808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829746.360129:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.360141:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829746.360151:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973191680:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829746.360152:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829746.360167:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973191808:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829746.360169:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829746.360175:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829746.360181:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6196 00000100:00080000:3.0:1787829746.360185:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829746.360189:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829746.360194:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829746.360197:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829746.360211:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973191936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829746.360216:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff9c1a7895cf00 x1874668973191808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829746.360243:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829746.360248:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829746.360250:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973191808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829746.360251:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829746.360262:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6196 00000100:00080000:3.0:1787829746.360270:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829746.360273:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829746.360275:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829746.360282:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787829746.360283:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829746.360290:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973191808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829746.360299:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973192064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829746.360307:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973191936:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829746.360375:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829746.360394:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff9c1a7895e940 x1874668973191936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829746.360412:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973191936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829746.360424:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.360428:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829746.360433:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973191936:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829746.360449:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973192064:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829746.360477:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829746.360493:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829746/real 1787829746] req@ffff9c1a7895c780 x1874668973192064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829746.360507:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973192064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829746.360514:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829746.360517:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829746.360519:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973192064:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829746.871413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829746.871440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829747.127504:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829747.895488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829747.895488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829748.151560:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829748.919458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829748.919459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829749.175567:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829749.943454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829749.943454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829750.199563:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829750.967482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829750.967483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829751.223557:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829751.479496:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829751.479504:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829751.479508:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829751.479517:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6201 00000100:00080000:0.0:1787829751.479527:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829751.479533:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829751.479570:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973192192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829751.479615:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829751.479635:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829751.479636:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829751.479640:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6201 00000100:00080000:0.0:1787829751.479644:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829751.479647:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829751.479655:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829751.479669:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973192320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829751.479678:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829751.479681:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829751.479683:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829751.479686:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6201 00000100:00080000:0.0:1787829751.479688:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829751.479693:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973192192:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829751.479695:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829751.479703:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829751.479714:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973192448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829751.479723:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829751.479726:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829751.479728:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829751.479731:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6201 00000100:00080000:0.0:1787829751.479734:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829751.479737:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829751.479740:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829751.479748:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973192576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829751.479754:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829751.479757:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829751.479760:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829751.479762:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829751.479766:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6201 00000100:00080000:0.0:1787829751.479770:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829751.479773:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829751.479777:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829751.479787:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973192704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829751.479794:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829751.479797:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829751.479799:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829751.479804:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6201 00000100:00100000:2.0:1787829751.479805:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829751/real 1787829751] req@ffff9c1a78860b40 x1874668973192192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829751.479807:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829751.479814:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829751.479838:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829751.479852:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973192192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829751.479854:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973192832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829751.479888:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829751.479898:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829751.479903:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829751.479907:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829751.479908:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6201 00000100:00080000:0.0:1787829751.479917:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829751.479920:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829751.479925:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829751.479931:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973192192:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829751.479933:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829751.479944:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973192320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829751.479949:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973192960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829751.479975:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829751.479993:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829751/real 1787829751] req@ffff9c1a78862d00 x1874668973192320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829751.480010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973192320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829751.480021:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829751.480026:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829751.480030:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973192320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829751.480057:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973192448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829751.480076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829751.480088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829751/real 1787829751] req@ffff9c1a78863840 x1874668973192448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829751.480103:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973192448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829751.480114:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829751.480118:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829751.480123:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973192448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829751.480132:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973192576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829751.480155:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829751.480165:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829751/real 1787829751] req@ffff9c1a788621c0 x1874668973192576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829751.480191:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973192576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829751.480198:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829751.480201:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829751.480203:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973192576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829751.480208:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973192704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829751.480221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829751.480228:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829751/real 1787829751] req@ffff9c1a78860f00 x1874668973192704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829751.480237:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973192704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829751.480242:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829751.480244:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829751.480247:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973192704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829751.480251:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973192832:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829751.480271:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829751.480284:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829751/real 1787829751] req@ffff9c1a78861a40 x1874668973192832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829751.480300:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973192832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829751.480311:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829751.480315:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829751.480318:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973192832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829751.480323:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973192960:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829751.480340:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829751.480349:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829751/real 1787829751] req@ffff9c1a78862940 x1874668973192960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829751.480357:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973192960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829751.480364:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829751.480366:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829751.480368:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973192960:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829751.991460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829751.991491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829752.247518:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829753.015417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829753.015418:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829753.271492:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829754.039417:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829754.039423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829754.295554:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829755.063435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829755.063435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829755.319533:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829756.087463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829756.087463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829756.343533:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829756.599469:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829756.599477:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829756.599480:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829756.599489:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6206 00000100:00080000:3.0:1787829756.599498:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829756.599504:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829756.599535:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973193088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829756.599582:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829756.599586:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829756.599589:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829756.599594:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6206 00000100:00080000:3.0:1787829756.599598:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829756.599602:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829756.599613:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829756.599628:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973193216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829756.599637:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973193088:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829756.599639:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829756.599645:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829756.599648:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829756.599652:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6206 00000100:00080000:3.0:1787829756.599654:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829756.599657:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829756.599661:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829756.599669:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973193344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829756.599686:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829756.599692:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829756.599693:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829756.599696:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6206 00000100:00080000:3.0:1787829756.599699:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829756.599704:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829756.599706:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829756.599710:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829756.599719:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973193472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829756.599727:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829756.599729:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829756.599730:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829756.599732:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6206 00000100:00080000:3.0:1787829756.599734:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829756.599736:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829756.599739:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829756.599749:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973193600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829756.599755:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829756/real 1787829756] req@ffff9c1a7895c3c0 x1874668973193088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829756.599756:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829756.599764:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829756.599767:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829756.599774:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6206 00000100:00080000:3.0:1787829756.599777:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829756.599780:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829756.599784:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829756.599785:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973193088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829756.599792:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973193728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829756.599816:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829756.599817:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829756.599823:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829756.599825:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829756.599827:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829756.599832:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973193088:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829756.599834:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6206 00000100:00080000:3.0:1787829756.599837:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829756.599839:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829756.599841:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829756.599849:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973193856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829756.599875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973193216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829756.599898:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829756.599908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829756/real 1787829756] req@ffff9c1a7895c780 x1874668973193216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829756.599917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973193216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829756.599923:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829756.599925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829756.599927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973193216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829756.599931:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973193344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829756.599942:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829756.599951:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829756/real 1787829756] req@ffff9c1a7895e940 x1874668973193344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829756.599962:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973193344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829756.599971:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829756.599974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829756.599978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973193344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829756.599984:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973193472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829756.600005:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829756.600014:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829756/real 1787829756] req@ffff9c1a7895cf00 x1874668973193472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829756.600022:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973193472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829756.600027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829756.600029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829756.600031:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973193472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829756.600035:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973193600:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829756.600047:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829756.600053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829756/real 1787829756] req@ffff9c1a7895e580 x1874668973193600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829756.600060:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973193600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829756.600065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829756.600067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829756.600068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973193600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829756.600072:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973193728:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829756.600085:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829756.600095:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829756/real 1787829756] req@ffff9c1a7895c000 x1874668973193728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829756.600108:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973193728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829756.600115:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829756.600117:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829756.600119:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973193728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829756.600124:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973193856:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829756.600138:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829756.600145:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829756/real 1787829756] req@ffff9c1a7895f480 x1874668973193856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829756.600152:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973193856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829756.600157:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829756.600159:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829756.600161:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973193856:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829757.111409:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829757.111430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829757.367475:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829758.135472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829758.135524:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829758.391598:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829759.159420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829759.159473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829759.415567:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829760.183498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829760.183498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829760.439544:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829761.207445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829761.207451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829761.463561:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829761.719469:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829761.719481:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829761.719487:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829761.719499:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6212 00000100:00080000:3.0:1787829761.719551:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829761.719557:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829761.719588:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973193984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829761.719610:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829761.719612:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829761.719614:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829761.719617:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6212 00000100:00080000:3.0:1787829761.719619:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829761.719622:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829761.719629:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829761.719638:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973194112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829761.719646:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829761.719649:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829761.719650:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829761.719654:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6212 00000100:00100000:2.0:1787829761.719657:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973193984:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829761.719657:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829761.719663:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829761.719668:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829761.719676:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973194240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829761.719684:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829761.719686:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829761.719688:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829761.719690:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6212 00000100:00080000:3.0:1787829761.719692:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829761.719694:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829761.719698:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829761.719704:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973194368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829761.719711:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829761.719713:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829761.719714:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829761.719719:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6212 00000800:00000100:2.0:1787829761.719721:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829761.719723:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829761.719727:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829761.719730:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829761.719738:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973194496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829761.719744:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829761.719746:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829761.719747:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829761.719750:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6212 00000100:00080000:3.0:1787829761.719753:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829761.719755:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829761.719759:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829761.719760:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829761/real 1787829761] req@ffff9c1a7895d2c0 x1874668973193984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829761.719768:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973194624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829761.719776:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829761.719778:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973193984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829761.719779:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829761.719783:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829761.719790:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6212 00000100:00080000:3.0:1787829761.719795:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829761.719800:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829761.719806:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829761.719812:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829761.719817:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973194752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829761.719819:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829761.719828:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973193984:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829761.719849:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973194112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829761.719865:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829761.719873:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829761/real 1787829761] req@ffff9c1a7895cb40 x1874668973194112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829761.719881:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973194112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829761.719887:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829761.719889:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829761.719891:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973194112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829761.719895:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973194240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829761.719905:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829761.719910:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829761/real 1787829761] req@ffff9c1a7895de00 x1874668973194240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829761.719917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973194240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829761.719922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829761.719924:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829761.719926:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973194240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829761.719930:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973194368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829761.719939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829761.719945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829761/real 1787829761] req@ffff9c1a7895f840 x1874668973194368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829761.719953:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973194368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829761.719958:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829761.719959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829761.719961:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973194368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829761.719964:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973194496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829761.719973:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829761.719978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829761/real 1787829761] req@ffff9c1a7895f0c0 x1874668973194496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829761.719985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973194496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829761.719990:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829761.719992:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829761.719994:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973194496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829761.719997:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973194624:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829761.720007:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829761.720028:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829761/real 1787829761] req@ffff9c1a7895fc00 x1874668973194624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829761.720041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973194624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829761.720050:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829761.720053:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829761.720056:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973194624:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829761.720062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973194752:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829761.720079:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829761.720088:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829761/real 1787829761] req@ffff9c1a7895e1c0 x1874668973194752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829761.720100:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973194752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829761.720108:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829761.720111:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829761.720114:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973194752:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829762.231452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829762.231453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829762.487534:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829763.255462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829763.255464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829763.511512:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829764.279456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829764.279482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829764.535523:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829765.303468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829765.303471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829765.559476:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829766.327450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829766.327450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829766.583491:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829766.839495:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829766.839505:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829766.839509:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829766.839521:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6217 00000100:00080000:3.0:1787829766.839534:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829766.839541:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829766.839584:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973194880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829766.839612:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829766.839615:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829766.839616:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829766.839620:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6217 00000100:00080000:3.0:1787829766.839622:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829766.839625:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829766.839635:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829766.839647:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973195008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829766.839655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829766.839659:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829766.839661:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829766.839663:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6217 00000100:00080000:3.0:1787829766.839665:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829766.839668:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829766.839671:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829766.839680:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973195136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829766.839692:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973194880:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829766.839695:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829766.839716:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829766.839717:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829766.839726:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6217 00000100:00080000:3.0:1787829766.839730:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829766.839734:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829766.839738:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829766.839749:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973195264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829766.839756:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829766.839758:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829766.839759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829766.839762:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6217 00000100:00080000:3.0:1787829766.839765:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829766.839767:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829766.839771:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829766.839779:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973195392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829766.839786:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829766.839790:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829766.839792:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829766.839799:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6217 00000100:00080000:3.0:1787829766.839805:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829766.839811:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829766.839819:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973195008:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829766.839820:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829766.839849:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973195520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829766.839860:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829766.839862:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829766.839864:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829766.839867:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6217 00000100:00080000:3.0:1787829766.839871:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829766.839875:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829766.839880:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973195136:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829766.839880:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829766.839904:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973195648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829766.839917:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973195264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829766.839938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973195392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829766.839958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973195520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829766.839976:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973195648:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829767.351506:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829767.351539:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829767.607577:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829768.375419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829768.375452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829768.631513:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829769.399507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829769.399524:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829769.655516:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829769.911601:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787829769.911640:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829769.911656:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829769.911692:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829769.911701:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829769.911709:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829769.911726:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829769.911739:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787829769.911751:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829766/real 1787829769] req@ffff9c1a7895fc00 x1874668973195136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829769.911752:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787829769.911780:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973195136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.911792:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.911797:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829769.911801:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973195136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829769.911809:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829766/real 1787829769] req@ffff9c1a7895f0c0 x1874668973195264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.911823:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973195264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.911832:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.911836:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829769.911840:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973195264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829769.911846:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829766/real 1787829769] req@ffff9c1a7895f840 x1874668973195392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.911855:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973195392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.911866:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.911870:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829769.911873:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973195392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829769.911881:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829766/real 1787829769] req@ffff9c1a7895de00 x1874668973195520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.911895:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973195520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.911903:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.911906:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829769.911909:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973195520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829769.911916:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829766/real 1787829769] req@ffff9c1a7895da40 x1874668973194880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.911928:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973194880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.911935:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.911939:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829769.911943:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973194880:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829769.911949:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829766/real 1787829769] req@ffff9c1a7895e1c0 x1874668973195008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.911961:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973195008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.911969:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.911973:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829769.911976:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973195008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829769.912012:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829766/real 1787829769] req@ffff9c1a7895cb40 x1874668973195648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829769.912029:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973195648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829769.912037:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829769.912040:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829769.912043:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973195648:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829770.423436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829770.423436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829770.679507:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829771.447487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829771.447487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829771.703564:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829771.959559:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829771.959573:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829771.959578:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829771.959593:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6222 00000100:00080000:0.0:1787829771.959608:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829771.959617:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829771.959664:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973195776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829771.959707:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829771.959712:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829771.959713:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829771.959719:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6222 00000100:00080000:0.0:1787829771.959723:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829771.959727:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829771.959740:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829771.959755:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973195904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829771.959765:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973195776:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829771.959769:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829771.959774:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829771.959777:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829771.959783:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6222 00000100:00080000:0.0:1787829771.959787:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829771.959791:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829771.959796:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829771.959812:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973196032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829771.959824:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829771.959828:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829771.959830:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829771.959834:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6222 00000100:00080000:0.0:1787829771.959839:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829771.959845:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829771.959850:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829771.959862:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829771.959866:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973196160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829771.959877:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829771.959880:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829771.959882:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829771.959886:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6222 00000100:00080000:0.0:1787829771.959891:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829771.959896:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829771.959902:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829771.959915:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973196288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829771.959925:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829771.959928:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829771.959930:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829771.959934:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829771/real 1787829771] req@ffff9c1a78860780 x1874668973195776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829771.959938:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6222 00000100:00080000:0.0:1787829771.959946:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829771.959961:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829771.959964:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973195776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829771.959970:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829771.959980:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829771.959987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829771.959992:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973196416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829771.959996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973195776:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829771.960020:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829771.960024:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829771.960026:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829771.960032:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6222 00000100:00080000:0.0:1787829771.960037:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829771.960040:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973195904:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829771.960046:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829771.960056:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829771.960072:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973196544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829771.960082:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829771.960103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829771/real 1787829771] req@ffff9c1a78861e00 x1874668973195904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829771.960117:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973195904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829771.960126:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829771.960130:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829771.960133:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973195904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829771.960139:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973196032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829771.960159:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829771.960168:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829771/real 1787829771] req@ffff9c1a788612c0 x1874668973196032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829771.960180:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973196032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829771.960189:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829771.960192:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829771.960196:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973196032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829771.960202:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973196160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829771.960221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829771.960233:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829771/real 1787829771] req@ffff9c1a78863c00 x1874668973196160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829771.960252:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973196160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829771.960261:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829771.960264:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829771.960267:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973196160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829771.960273:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973196288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829771.960294:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829771.960306:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829771/real 1787829771] req@ffff9c1a788630c0 x1874668973196288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829771.960319:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973196288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829771.960328:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829771.960333:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829771.960336:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973196288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829771.960343:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973196416:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829771.960400:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829771.960412:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829771/real 1787829771] req@ffff9c1a78862580 x1874668973196416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829771.960425:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973196416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829771.960458:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829771.960462:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829771.960465:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973196416:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829771.960488:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973196544:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829771.960511:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829771.960524:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829771/real 1787829771] req@ffff9c1a78861680 x1874668973196544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829771.960536:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973196544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829771.960545:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829771.960554:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829771.960557:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973196544:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829772.471476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829772.471476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829772.727536:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829773.495499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787829773.495515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829773.751524:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829774.519503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829774.519635:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829774.775551:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829775.543552:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829775.543562:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829775.799520:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829776.567458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829776.567562:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829776.823560:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829777.079436:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.079443:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829777.079447:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.079454:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6227 00000100:00080000:3.0:1787829777.079462:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829777.079467:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829777.079496:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973196672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829777.079516:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.079519:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829777.079520:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.079523:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6227 00000100:00080000:3.0:1787829777.079525:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829777.079527:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829777.079533:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829777.079542:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973196800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829777.079549:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.079551:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829777.079553:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.079557:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6227 00000100:00100000:2.0:1787829777.079558:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973196672:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829777.079561:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829777.079567:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829777.079571:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829777.079585:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973196928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829777.079595:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.079598:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829777.079600:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.079603:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6227 00000100:00080000:3.0:1787829777.079606:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829777.079610:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787829777.079614:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829777.079617:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829777.079628:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973197056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829777.079634:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.079636:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829777.079637:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.079640:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6227 00000100:00080000:3.0:1787829777.079643:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829777.079646:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829777.079650:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff9c1a7895d2c0 x1874668973196672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829777.079650:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829777.079666:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973196672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829777.079670:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973197184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829777.079681:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829777.079682:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.079687:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829777.079689:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829777.079689:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829777.079697:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973196672:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829777.079697:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6227 00000100:00080000:3.0:1787829777.079703:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829777.079706:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829777.079709:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829777.079720:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973196800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829777.079721:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973197312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829777.079730:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829777.079733:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829777.079735:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829777.079737:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6227 00000100:00080000:3.0:1787829777.079740:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829777.079741:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829777.079745:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829777.079749:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829777.079753:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff9c1a7895f480 x1874668973196800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829777.079759:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973197440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829777.079773:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973196800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.079780:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.079783:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829777.079786:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973196800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829777.079791:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973196928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829777.079808:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829777.079817:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff9c1a7895c000 x1874668973196928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829777.079829:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973196928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.079837:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.079840:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829777.079844:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973196928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829777.079849:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973197056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829777.079863:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829777.079870:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff9c1a7895e580 x1874668973197056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829777.079877:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973197056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.079882:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.079883:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829777.079885:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973197056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829777.079888:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973197184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829777.079897:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829777.079911:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff9c1a7895cf00 x1874668973197184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829777.079917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973197184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.079922:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.079924:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829777.079926:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973197184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829777.079938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973197312:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829777.079954:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829777.079960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff9c1a7895e940 x1874668973197312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829777.079971:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973197312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.079976:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.079977:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829777.079979:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973197312:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829777.079982:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973197440:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829777.079991:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829777.079996:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829777/real 1787829777] req@ffff9c1a7895c780 x1874668973197440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829777.080002:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973197440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829777.080006:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829777.080008:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829777.080009:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973197440:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829777.591454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829777.591455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829777.847504:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829778.615465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829778.615473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829778.871548:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829779.639461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829779.639481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829779.895513:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829780.663474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829780.663474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829780.919525:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829781.687449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829781.687449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829781.943511:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829782.199519:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829782.199528:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829782.199532:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829782.199539:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6232 00000100:00080000:0.0:1787829782.199548:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829782.199553:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829782.199579:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973197568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829782.199599:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829782.199601:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829782.199602:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829782.199605:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6232 00000100:00080000:0.0:1787829782.199608:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829782.199610:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829782.199618:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829782.199628:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973197696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829782.199636:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973197568:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829782.199639:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829782.199646:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829782.199648:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829782.199652:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6232 00000100:00080000:0.0:1787829782.199656:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829782.199660:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829782.199665:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829782.199688:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973197824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829782.199696:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829782.199703:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829782.199706:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829782.199708:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829782.199711:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6232 00000100:00080000:0.0:1787829782.199714:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829782.199717:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829782.199724:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829782.199731:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973197952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829782.199741:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829782.199743:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829782.199744:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829782.199746:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6232 00000100:00080000:0.0:1787829782.199749:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829782.199752:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829782.199756:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829782.199767:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973198080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829782.199775:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829782.199778:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829782.199780:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829782.199784:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff9c1a78860000 x1874668973197568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829782.199787:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6232 00000100:00080000:0.0:1787829782.199792:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829782.199799:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829782.199806:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829782.199816:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973198208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829782.199823:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829782.199825:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829782.199826:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829782.199828:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6232 00000100:00080000:0.0:1787829782.199831:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829782.199833:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829782.199836:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829782.199842:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973198336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829782.199911:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973197568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829782.199921:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829782.199963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829782.199968:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973197568:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829782.199989:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973197696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829782.200009:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829782.200019:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff9c1a78861680 x1874668973197696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829782.200027:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973197696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829782.200032:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829782.200034:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829782.200036:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973197696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829782.200040:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973197824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829782.200057:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829782.200063:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff9c1a78862580 x1874668973197824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829782.200070:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973197824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829782.200075:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829782.200076:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829782.200078:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973197824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829782.200081:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973197952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829782.200090:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829782.200095:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff9c1a788630c0 x1874668973197952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829782.200101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973197952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829782.200106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829782.200108:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829782.200110:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973197952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829782.200113:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973198080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787829782.200121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829782.200125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff9c1a78863c00 x1874668973198080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829782.200131:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973198080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829782.200136:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829782.200138:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829782.200139:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973198080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829782.200142:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973198208:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829782.200151:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829782.200155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff9c1a788612c0 x1874668973198208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829782.200161:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973198208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829782.200166:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829782.200167:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829782.200169:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973198208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829782.200172:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973198336:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787829782.200180:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829782.200185:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829782/real 1787829782] req@ffff9c1a78861e00 x1874668973198336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829782.200190:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973198336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829782.200195:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829782.200196:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829782.200198:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973198336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829782.711457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829782.711484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829782.967489:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829783.735439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829783.735444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829783.991508:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829784.759451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829784.759451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829785.015449:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829785.783418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829785.783720:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829786.039482:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829786.807441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829786.807442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829787.063513:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829787.319451:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829787.319459:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829787.319462:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829787.319470:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6237 00000100:00080000:3.0:1787829787.319479:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829787.319484:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829787.319519:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973198464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829787.319542:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829787.319545:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829787.319546:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829787.319549:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6237 00000100:00080000:3.0:1787829787.319551:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829787.319553:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829787.319561:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829787.319572:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973198592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829787.319579:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829787.319582:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829787.319585:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829787.319587:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973198464:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829787.319590:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6237 00000100:00080000:3.0:1787829787.319596:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829787.319599:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829787.319602:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829787.319610:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973198720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829787.319618:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829787.319621:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829787.319622:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829787.319624:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6237 00000100:00080000:3.0:1787829787.319637:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829787.319639:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829787.319643:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829787.319647:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829787.319652:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973198848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829787.319661:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829787.319663:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829787.319665:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829787.319667:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6237 00000100:00080000:3.0:1787829787.319669:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829787.319671:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829787.319675:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829787.319683:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff9c1a7895c3c0 x1874668973198464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829787.319684:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973198976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829787.319715:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829787.319717:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829787.319718:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829787.319721:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973198464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829787.319724:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6237 00000100:00080000:3.0:1787829787.319729:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829787.319736:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829787.319738:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829787.319740:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829787.319744:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829787.319748:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973198464:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829787.319752:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973199104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829787.319764:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829787.319766:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829787.319767:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829787.319770:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6237 00000100:00100000:2.0:1787829787.319773:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973198592:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829787.319775:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829787.319781:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829787.319789:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829787.319798:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973199232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829787.319805:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829787.319827:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff9c1a7895c780 x1874668973198592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.319843:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973198592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.319850:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.319852:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829787.319854:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973198592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829787.319859:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973198720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829787.319884:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829787.319892:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff9c1a7895e940 x1874668973198720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.319900:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973198720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.319909:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.319911:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829787.319913:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973198720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829787.319917:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973198848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829787.319928:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829787.319934:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff9c1a7895cf00 x1874668973198848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.319941:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973198848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.319946:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.319948:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829787.319950:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973198848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829787.319954:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973198976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829787.319971:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829787.319978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff9c1a7895e580 x1874668973198976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.319985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973198976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.319990:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.319992:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829787.319995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973198976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829787.319999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973199104:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829787.320014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829787.320021:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff9c1a7895c000 x1874668973199104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.320033:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973199104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.320041:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.320044:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829787.320047:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973199104:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829787.320068:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973199232:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829787.320087:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829787.320095:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829787/real 1787829787] req@ffff9c1a7895f480 x1874668973199232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829787.320103:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973199232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829787.320108:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829787.320110:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829787.320112:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973199232:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829787.831425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829787.831426:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829788.087525:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829788.855454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829788.855454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829789.111512:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829789.879466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829789.879466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829790.135504:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829790.903495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829790.903500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829791.159633:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829791.927432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829791.927432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829792.183533:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829792.439460:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829792.439469:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829792.439472:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829792.439481:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6242 00000100:00080000:3.0:1787829792.439490:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829792.439495:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829792.439527:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973199360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829792.439561:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829792.439565:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829792.439566:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829792.439569:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6242 00000100:00080000:3.0:1787829792.439571:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829792.439574:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829792.439582:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829792.439593:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973199488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829792.439605:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829792.439608:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829792.439611:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829792.439615:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6242 00000100:00080000:3.0:1787829792.439618:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829792.439623:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829792.439629:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829792.439644:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973199616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829792.439649:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973199360:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829792.439657:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829792.439667:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829792.439668:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829792.439672:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6242 00000100:00080000:3.0:1787829792.439675:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829792.439678:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829792.439683:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829792.439692:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973199744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829792.439699:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829792.439702:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829792.439703:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829792.439706:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6242 00000100:00080000:3.0:1787829792.439709:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829792.439711:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829792.439713:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829792.439721:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973199872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829792.439727:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829792.439729:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829792.439730:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829792.439733:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6242 00000100:00080000:3.0:1787829792.439743:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829792.439745:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829792.439748:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829792.439757:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973200000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829792.439763:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829792.439772:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829792.439772:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829792.439777:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829792.439795:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6242 00000100:00080000:3.0:1787829792.439799:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829792.439802:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829792.439806:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829792.439813:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973200128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829792.439859:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff9c1a7895d2c0 x1874668973199360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.439884:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973199360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.439898:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.439904:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829792.439909:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973199360:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829792.439919:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973199488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829792.439950:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829792.439965:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff9c1a7895cb40 x1874668973199488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.439981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973199488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.439991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.439995:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829792.439999:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973199488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829792.440041:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973199616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829792.440066:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829792.440080:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff9c1a7895e1c0 x1874668973199616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.440093:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973199616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.440110:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.440114:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829792.440118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973199616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829792.440126:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973199744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829792.440148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829792.440162:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff9c1a7895da40 x1874668973199744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.440176:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973199744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.440187:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.440191:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829792.440194:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973199744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829792.440202:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973199872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829792.440225:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829792.440238:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff9c1a7895de00 x1874668973199872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.440251:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973199872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.440261:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.440264:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829792.440269:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973199872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829792.440276:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973200000:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829792.440316:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829792.440330:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff9c1a7895f840 x1874668973200000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.440344:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973200000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.440401:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.440405:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829792.440409:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973200000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829792.440417:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973200128:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829792.440441:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829792.440454:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829792/real 1787829792] req@ffff9c1a7895f0c0 x1874668973200128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829792.440467:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973200128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829792.440477:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829792.440480:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829792.440484:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973200128:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829792.951473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829792.951473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829793.207623:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829793.975441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829793.975441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829794.231496:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829794.999473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829794.999483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829795.255572:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829796.023443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829796.023473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829796.279564:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829797.047477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829797.047477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829797.303579:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829797.559453:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829797.559468:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829797.559474:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829797.559486:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6247 00000100:00080000:0.0:1787829797.559499:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829797.559506:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829797.559549:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973200256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829797.559578:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829797.559583:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829797.559585:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829797.559591:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6247 00000100:00080000:0.0:1787829797.559596:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829797.559603:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829797.559616:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829797.559620:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973200256:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829797.559634:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973200384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829797.559646:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829797.559651:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829797.559653:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829797.559657:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6247 00000100:00080000:0.0:1787829797.559660:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829797.559663:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829797.559667:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829797.559677:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973200512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829797.559686:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829797.559689:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829797.559690:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829797.559693:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6247 00000800:00000100:2.0:1787829797.559697:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829797.559699:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829797.559705:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829797.559709:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829797.559718:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973200640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829797.559725:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829797.559728:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829797.559729:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829797.559732:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6247 00000100:00080000:0.0:1787829797.559735:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829797.559738:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829797.559742:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829797.559747:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829797/real 1787829797] req@ffff9c1a78860780 x1874668973200256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829797.559754:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973200768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829797.559774:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829797.559777:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829797.559779:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829797.559782:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6247 00000100:00080000:0.0:1787829797.559785:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829797.559788:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973200256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829797.559790:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829797.559796:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829797.559808:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973200896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829797.559810:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829797.559820:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829797.559822:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829797.559826:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973200256:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829797.559827:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829797.559831:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829797.559837:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6247 00000100:00080000:0.0:1787829797.559839:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829797.559842:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829797.559845:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829797.559851:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973201024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829797.559859:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973200384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829797.559882:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829797.559893:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829797/real 1787829797] req@ffff9c1a78862940 x1874668973200384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829797.559902:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973200384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829797.559908:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829797.559910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829797.559913:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973200384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829797.559917:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973200512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829797.559928:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829797.559934:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829797/real 1787829797] req@ffff9c1a78861a40 x1874668973200512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829797.559942:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973200512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829797.559952:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829797.559954:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829797.559956:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973200512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829797.559959:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973200640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829797.559970:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829797.559975:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829797/real 1787829797] req@ffff9c1a78860f00 x1874668973200640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829797.559982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973200640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829797.559988:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829797.559990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829797.559992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973200640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829797.559996:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973200768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829797.560013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829797.560024:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829797/real 1787829797] req@ffff9c1a788621c0 x1874668973200768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829797.560038:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973200768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829797.560047:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829797.560050:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829797.560055:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973200768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829797.560059:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973200896:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829797.560076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829797.560084:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829797/real 1787829797] req@ffff9c1a78863840 x1874668973200896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829797.560092:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973200896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829797.560098:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829797.560100:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829797.560102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973200896:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829797.560116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973201024:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829797.560127:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829797.560132:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829797/real 1787829797] req@ffff9c1a78862d00 x1874668973201024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829797.560140:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973201024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829797.560145:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829797.560147:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829797.560149:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973201024:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829798.071441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829798.071441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829798.327540:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829799.095466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829799.095472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829799.351579:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829800.119442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829800.119442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829800.375528:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829801.143471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829801.143472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829801.399556:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829802.167481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829802.167481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829802.423483:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829802.679516:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829802.679532:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829802.679538:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829802.679551:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6253 00000100:00080000:3.0:1787829802.679565:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829802.679575:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829802.679625:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973201152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829802.679668:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829802.679673:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829802.679675:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829802.679680:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6253 00000100:00080000:3.0:1787829802.679685:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829802.679690:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829802.679702:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829802.679720:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973201280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829802.679736:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973201152:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829802.679756:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829802.679760:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829802.679763:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829802.679767:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6253 00000100:00080000:3.0:1787829802.679772:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829802.679777:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829802.679783:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829802.679797:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973201408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829802.679810:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829802.679814:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829802.679816:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829802.679820:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6253 00000100:00080000:3.0:1787829802.679825:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829802.679830:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829802.679835:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829802.679843:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829802.679850:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973201536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829802.679862:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829802.679866:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829802.679868:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829802.679873:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6253 00000100:00080000:3.0:1787829802.679877:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829802.679882:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829802.679888:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829802.679891:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829802/real 1787829802] req@ffff9c1a7895fc00 x1874668973201152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829802.679902:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973201664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829802.679913:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829802.679916:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829802.679918:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829802.679923:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6253 00000100:00080000:3.0:1787829802.679928:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829802.679933:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973201152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829802.679937:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829802.679946:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829802.679953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829802.679961:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973201792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829802.679970:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829802.679986:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829802.679991:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829802.679992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973201152:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829802.679996:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829802.680003:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6253 00000100:00080000:3.0:1787829802.680007:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829802.680012:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829802.680018:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829802.680032:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973201920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829802.680044:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973201280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829802.680074:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829802.680096:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829802/real 1787829802] req@ffff9c1a7895f0c0 x1874668973201280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829802.680111:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973201280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829802.680121:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829802.680125:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829802.680129:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973201280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829802.680136:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973201408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829802.680158:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829802.680167:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829802/real 1787829802] req@ffff9c1a7895f840 x1874668973201408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829802.680176:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973201408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829802.680186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829802.680189:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829802.680191:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973201408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829802.680195:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973201536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829802.680242:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829802.680252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829802/real 1787829802] req@ffff9c1a7895de00 x1874668973201536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829802.680260:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973201536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829802.680267:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829802.680269:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829802.680271:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973201536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829802.680275:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973201664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829802.680305:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829802.680313:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829802/real 1787829802] req@ffff9c1a7895da40 x1874668973201664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829802.680321:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973201664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829802.680326:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829802.680329:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829802.680371:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973201664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829802.680377:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973201792:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829802.680392:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829802.680421:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829802/real 1787829802] req@ffff9c1a7895e1c0 x1874668973201792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829802.680435:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973201792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829802.680444:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829802.680448:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829802.680452:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973201792:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829802.680491:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973201920:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829802.680508:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829802.680516:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829802/real 1787829802] req@ffff9c1a7895cb40 x1874668973201920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829802.680524:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973201920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829802.680530:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829802.680532:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829802.680534:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973201920:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829803.191454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829803.191487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829803.447520:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829804.215475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829804.215479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829804.471533:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829805.239529:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829805.239529:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829805.495576:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829806.263478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829806.263488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829806.519550:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829807.287436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829807.287436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829807.543525:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829807.799455:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829807.799466:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829807.799471:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829807.799482:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6258 00000100:00080000:3.0:1787829807.799495:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829807.799508:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829807.799552:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973202048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829807.799578:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829807.799582:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829807.799584:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829807.799587:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6258 00000100:00080000:3.0:1787829807.799590:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829807.799593:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829807.799603:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829807.799613:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973202176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829807.799621:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829807.799624:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829807.799627:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829807.799629:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973202048:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829807.799632:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6258 00000100:00080000:3.0:1787829807.799637:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829807.799641:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829807.799644:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829807.799651:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973202304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829807.799659:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829807.799662:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829807.799663:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829807.799666:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6258 00000100:00080000:3.0:1787829807.799668:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829807.799670:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829807.799675:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829807.799682:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973202432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829807.799689:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829807.799691:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829807.799693:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829807.799695:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6258 00000100:00080000:3.0:1787829807.799698:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829807.799700:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829807.799702:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829807.799707:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829807.799712:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973202560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829807.799719:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829807.799722:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829807.799723:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829807.799726:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6258 00000100:00080000:3.0:1787829807.799739:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829807.799742:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829807.799746:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829807.799755:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973202688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829807.799763:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829807.799764:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829807/real 1787829807] req@ffff9c1a7895d2c0 x1874668973202048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829807.799767:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829807.799770:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829807.799777:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6258 00000100:00080000:3.0:1787829807.799782:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829807.799787:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829807.799790:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829807.799798:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973202816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829807.799809:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973202048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829807.799820:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829807.799824:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829807.799828:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973202048:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829807.799859:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973202176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829807.799880:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829807.799889:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829807/real 1787829807] req@ffff9c1a7895f480 x1874668973202176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829807.799898:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973202176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829807.799911:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829807.799914:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829807.799916:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973202176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829807.799920:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973202304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829807.799933:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829807.799940:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829807/real 1787829807] req@ffff9c1a7895c000 x1874668973202304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829807.799947:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973202304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829807.799957:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829807.799959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829807.799962:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973202304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829807.799965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973202432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829807.799975:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829807.799981:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829807/real 1787829807] req@ffff9c1a7895e580 x1874668973202432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829807.799988:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973202432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829807.799994:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829807.799995:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829807.799997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973202432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829807.800001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973202560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829807.800010:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829807.800016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829807/real 1787829807] req@ffff9c1a7895cf00 x1874668973202560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829807.800022:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973202560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829807.800027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829807.800029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829807.800032:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973202560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829807.800036:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973202688:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829807.800046:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829807.800052:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829807/real 1787829807] req@ffff9c1a7895e940 x1874668973202688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829807.800059:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973202688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829807.800065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829807.800067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829807.800068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973202688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829807.800071:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973202816:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829807.800081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829807.800086:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829807/real 1787829807] req@ffff9c1a7895c780 x1874668973202816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829807.800093:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973202816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829807.800098:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829807.800100:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829807.800102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973202816:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829808.311488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829808.311493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829808.567569:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829809.335454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829809.335454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829809.591559:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829810.359454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829810.359454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829810.615501:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829811.383469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829811.383475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829811.639511:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829812.407471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829812.407471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829812.663614:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829812.919477:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829812.919488:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829812.919492:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829812.919503:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6263 00000100:00080000:0.0:1787829812.919514:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829812.919520:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829812.919555:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973202944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829812.919583:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829812.919587:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829812.919589:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829812.919593:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6263 00000100:00080000:0.0:1787829812.919597:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829812.919602:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829812.919613:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829812.919631:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973203072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829812.919643:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829812.919647:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829812.919649:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829812.919653:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6263 00000100:00080000:0.0:1787829812.919658:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829812.919663:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829812.919671:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829812.919674:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973202944:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829812.919688:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973203200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829812.919710:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829812.919715:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829812.919717:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829812.919722:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6263 00000100:00080000:0.0:1787829812.919728:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829812.919732:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829812.919738:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829812.919751:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973203328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829812.919761:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829812.919765:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1787829812.919771:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829812.919773:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829812.919781:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6263 00000100:00080000:0.0:1787829812.919787:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829812.919793:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829812.919801:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829812.919818:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973203456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829812.919830:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829812/real 1787829812] req@ffff9c1a78860b40 x1874668973202944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829812.919831:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829812.919836:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829812.919844:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829812.919862:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6263 00000100:00080000:0.0:1787829812.919870:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829812.919875:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973202944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829812.919879:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787829812.919899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829812.919900:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829812.919907:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829812.919912:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973202944:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829812.919921:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973203584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829812.919922:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973203072:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829812.919947:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829812.919952:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829812.919955:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829812.919957:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829812.919969:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6263 00000100:00080000:0.0:1787829812.919976:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829812.919977:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829812/real 1787829812] req@ffff9c1a78862d00 x1874668973203072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829812.919990:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829812.919996:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973203072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829812.920001:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829812.920005:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829812.920010:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829812.920013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973203072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829812.920023:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973203712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829812.920047:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973203200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829812.920067:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829812.920076:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829812/real 1787829812] req@ffff9c1a78863840 x1874668973203200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829812.920084:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973203200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829812.920095:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829812.920098:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829812.920102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973203200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829812.920109:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973203328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829812.920129:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829812.920142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829812/real 1787829812] req@ffff9c1a788621c0 x1874668973203328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829812.920174:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973203328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829812.920184:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829812.920188:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829812.920192:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973203328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829812.920200:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973203456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829812.920221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829812.920230:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829812/real 1787829812] req@ffff9c1a78860f00 x1874668973203456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829812.920238:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973203456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829812.920244:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829812.920247:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829812.920249:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973203456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829812.920254:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973203584:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829812.920266:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829812.920272:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829812/real 1787829812] req@ffff9c1a78861a40 x1874668973203584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829812.920280:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973203584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829812.920285:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829812.920287:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829812.920289:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973203584:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829812.920293:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973203712:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829812.920303:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829812.920309:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829812/real 1787829812] req@ffff9c1a78862940 x1874668973203712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829812.920316:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973203712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829812.920321:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829812.920323:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829812.920325:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973203712:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829813.431432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829813.431461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829813.687564:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829814.455451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829814.455454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829814.711592:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829815.479407:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829815.479408:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829815.735612:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829816.503471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829816.503471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829816.759573:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829817.527412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829817.527477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829817.783542:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829818.039428:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829818.039439:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829818.039443:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829818.039454:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6268 00000100:00080000:3.0:1787829818.039465:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829818.039473:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829818.039507:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973203840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829818.039540:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829818.039543:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829818.039545:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829818.039550:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6268 00000100:00080000:3.0:1787829818.039554:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829818.039558:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829818.039566:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829818.039572:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973203840:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829818.039581:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973203968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829818.039591:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829818.039594:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829818.039596:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829818.039599:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6268 00000100:00080000:3.0:1787829818.039602:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829818.039604:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829818.039607:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829818.039616:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973204096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829818.039623:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829818.039626:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787829818.039630:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829818.039631:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829818.039637:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6268 00000100:00080000:3.0:1787829818.039641:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829818.039645:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829818.039648:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829818.039658:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973204224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829818.039664:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829818.039667:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829818.039668:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829818.039670:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6268 00000100:00080000:3.0:1787829818.039672:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829818.039674:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829818.039678:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829818.039690:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973204352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829818.039693:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff9c1a7895c3c0 x1874668973203840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829818.039714:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829818.039720:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829818.039722:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829818.039728:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973203840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829818.039729:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6268 00000100:00080000:3.0:1787829818.039738:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829818.039741:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829818.039746:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787829818.039749:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787829818.039754:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829818.039758:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973203840:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829818.039769:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973204480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829818.039781:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829818.039783:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829818.039785:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829818.039789:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973203968:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829818.039790:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6268 00000100:00080000:3.0:1787829818.039796:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829818.039815:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829818.039824:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff9c1a7895c780 x1874668973203968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829818.039833:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973203968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829818.039839:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829818.039848:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829818.039851:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973203968:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829818.039859:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829818.039863:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973204096:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829818.039866:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829818.039881:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973204608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829818.039894:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829818.039912:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff9c1a7895e940 x1874668973204096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829818.039923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973204096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829818.039928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829818.039931:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829818.039934:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973204096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829818.039938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973204224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829818.039953:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829818.039960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff9c1a7895cf00 x1874668973204224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829818.039968:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973204224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829818.039974:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829818.039976:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829818.039978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973204224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829818.039982:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973204352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829818.039994:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829818.040000:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff9c1a7895e580 x1874668973204352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829818.040008:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973204352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829818.040013:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829818.040015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829818.040017:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973204352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829818.040021:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973204480:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829818.040033:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829818.040041:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff9c1a7895c000 x1874668973204480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829818.040049:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973204480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829818.040063:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829818.040068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829818.040072:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973204480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829818.040103:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973204608:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829818.040148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829818.040171:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829818/real 1787829818] req@ffff9c1a7895f480 x1874668973204608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829818.040184:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973204608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829818.040192:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829818.040196:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829818.040198:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973204608:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829818.551453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829818.551461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829818.807601:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829819.575464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829819.575464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829819.831597:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829820.599449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829820.599450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829820.855596:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829821.623452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829821.623452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829821.879602:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829822.647440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829822.647440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829822.903607:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829823.159531:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829823.159538:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829823.159541:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829823.159549:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6273 00000100:00080000:0.0:1787829823.159557:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829823.159562:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829823.159591:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973204736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829823.159615:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829823.159618:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829823.159619:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829823.159622:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6273 00000100:00080000:0.0:1787829823.159624:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829823.159626:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829823.159632:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829823.159640:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973204864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829823.159648:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829823.159650:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829823.159652:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829823.159655:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6273 00000100:00080000:0.0:1787829823.159657:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829823.159659:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829823.159662:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829823.159667:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973204992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829823.159674:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829823.159676:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829823.159677:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829823.159679:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6273 00000100:00080000:0.0:1787829823.159681:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829823.159690:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829823.159692:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973204736:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829823.159694:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829823.159704:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973205120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829823.159711:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829823.159712:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829823.159713:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829823.159715:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6273 00000100:00080000:0.0:1787829823.159717:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829823.159719:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829823.159721:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829823.159729:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973205248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829823.159733:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829823.159735:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829823.159736:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829823.159738:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6273 00000100:00080000:0.0:1787829823.159741:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829823.159743:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829823.159746:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829823.159754:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973205376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829823.159762:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829823.159765:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829823.159767:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829823.159773:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6273 00000100:00080000:0.0:1787829823.159777:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829823.159780:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829823.159786:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829823.159789:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829823.159802:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973205504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829823.159857:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff9c1a78860780 x1874668973204736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829823.159887:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973204736/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829823.159899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829823.159904:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829823.159909:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973204736:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829823.159918:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973204864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829823.159943:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829823.159957:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff9c1a78861e00 x1874668973204864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829823.159969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973204864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829823.159977:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829823.159980:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829823.159984:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973204864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829823.160013:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973204992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829823.160032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829823.160042:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff9c1a788612c0 x1874668973204992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829823.160053:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973204992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829823.160067:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829823.160070:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829823.160073:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973204992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829823.160079:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973205120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829823.160113:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829823.160124:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff9c1a78863c00 x1874668973205120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829823.160134:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973205120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829823.160143:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829823.160146:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829823.160149:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973205120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829823.160155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973205248:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829823.160172:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829823.160182:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff9c1a788630c0 x1874668973205248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829823.160193:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973205248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829823.160201:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829823.160204:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829823.160208:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973205248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829823.160214:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973205376:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829823.160233:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829823.160243:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff9c1a78862580 x1874668973205376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829823.160252:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973205376/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829823.160259:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829823.160261:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829823.160273:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973205376:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829823.160280:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973205504:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829823.160297:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829823.160307:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829823/real 1787829823] req@ffff9c1a78861680 x1874668973205504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829823.160316:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973205504/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829823.160323:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829823.160326:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829823.160329:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973205504:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829823.671479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829823.671480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829823.927473:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829824.695421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829824.695422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829824.951963:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829825.719436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829825.719436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829825.975521:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829826.743410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829826.743410:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829826.999489:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829827.767442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829827.767442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829828.023501:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829828.279438:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829828.279446:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829828.279450:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829828.279458:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6278 00000100:00080000:2.0:1787829828.279466:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829828.279471:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829828.279505:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973205632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829828.279530:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829828.279534:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829828.279535:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829828.279540:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6278 00000100:00080000:2.0:1787829828.279544:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829828.279548:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829828.279557:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829828.279563:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973205632:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829828.279572:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973205760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829828.279591:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829828.279593:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829828.279594:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829828.279597:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6278 00000100:00080000:2.0:1787829828.279601:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829828.279603:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829828.279607:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829828.279615:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973205888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829828.279626:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829828.279634:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829828.279638:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829828.279640:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829828.279645:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6278 00000100:00080000:2.0:1787829828.279649:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829828.279654:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829828.279659:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829828.279666:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff9c1a49d8cb40 x1874668973205632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829828.279673:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973206016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829828.279682:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829828.279685:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973205632/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829828.279686:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829828.279689:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829828.279697:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6278 00000100:00080000:0.0:1787829828.279701:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829828.279702:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829828.279709:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829828.279710:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829828.279715:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973205632:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787829828.279715:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829828.279733:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973206144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829828.279739:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973205760:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829828.279742:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829828.279745:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829828.279747:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829828.279750:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6278 00000100:00080000:2.0:1787829828.279754:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829828.279757:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787829828.279761:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:2.0:1787829828.279761:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829828.279773:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973206272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829828.279774:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff9c1a49d8da40 x1874668973205760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829828.279789:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829828.279792:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829828.279796:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829828.279804:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6278 00000100:00100000:0.0:1787829828.279807:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973205760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829828.279809:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829828.279818:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829828.279827:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829828.279828:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829828.279833:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829828.279836:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973205760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829828.279840:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973205888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829828.279855:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973206400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829828.279856:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829828.279865:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff9c1a49d8de00 x1874668973205888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829828.279876:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973205888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829828.279882:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829828.279885:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829828.279887:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973205888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829828.279890:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973206016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829828.279900:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829828.279910:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff9c1a49d8c780 x1874668973206016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829828.279920:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973206016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829828.279928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829828.279931:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829828.279934:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973206016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829828.279950:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973206144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787829828.279970:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829828.279981:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff9c1a49d8c000 x1874668973206144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829828.279993:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973206144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829828.280000:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829828.280003:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829828.280006:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973206144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829828.280011:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973206272:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829828.280030:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829828.280040:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff9c1a49d8cf00 x1874668973206272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829828.280064:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973206272/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829828.280072:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829828.280075:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829828.280078:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973206272:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787829828.280084:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973206400:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787829828.280101:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829828.280111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829828/real 1787829828] req@ffff9c1a49d8cb40 x1874668973206400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829828.280123:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973206400/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829828.280130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829828.280133:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829828.280136:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973206400:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829828.791421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829828.791428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829829.047476:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829829.815493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829829.815494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829830.071578:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829830.839468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829830.839474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829831.095490:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829831.863489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829831.863489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829832.119538:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829832.887437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829832.887437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829833.143464:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829833.399455:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829833.399463:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829833.399468:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829833.399476:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6283 00000100:00080000:0.0:1787829833.399486:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829833.399491:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829833.399523:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973206528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829833.399549:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829833.399552:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829833.399554:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829833.399558:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6283 00000100:00080000:0.0:1787829833.399562:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829833.399567:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829833.399575:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829833.399582:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973206528:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829833.399590:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973206656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829833.399598:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829833.399601:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829833.399602:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829833.399605:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6283 00000100:00080000:0.0:1787829833.399607:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829833.399610:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829833.399613:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829833.399620:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973206784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829833.399627:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829833.399629:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829833.399630:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829833.399632:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6283 00000100:00080000:0.0:1787829833.399634:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829833.399636:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829833.399639:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829833.399645:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973206912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829833.399651:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829833.399653:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829833.399654:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829833.399656:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6283 00000100:00080000:0.0:1787829833.399659:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829833.399661:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829833.399664:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829833.399672:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973206656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829833.399673:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973207040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829833.399684:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829833.399686:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829833.399687:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829833.399689:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6283 00000100:00080000:0.0:1787829833.399691:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829833.399694:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829833.399697:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829833.399705:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973207168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829833.399709:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973206784:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829833.399713:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829833.399717:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829833.399718:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829833.399721:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6283 00000100:00080000:0.0:1787829833.399724:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829833.399726:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829833.399729:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829833.399737:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973206912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829833.399738:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973207296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829833.399767:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973207040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829833.399787:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973207168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829833.399809:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973207296:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829833.911443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829833.911445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829834.167506:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829834.935440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829834.935440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829835.191542:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829835.959452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829835.959467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829836.215524:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829836.471581:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:3.0:1787829836.471615:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829836.471627:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829836.471657:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829836.471662:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829836.471666:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829836.471672:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829836.471678:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787829836.471684:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787829836.471720:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829836] req@ffff9c1a78863c00 x1874668973207040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.471742:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973207040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.471753:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.471757:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829836.471761:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973207040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.471768:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829836] req@ffff9c1a788612c0 x1874668973207168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.471779:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973207168/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.471787:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.471790:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829836.471793:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973207168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829836.471799:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829836] req@ffff9c1a78861e00 x1874668973207296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.471811:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973207296/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.471818:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.471822:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829836.471826:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973207296:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829836.471832:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829836] req@ffff9c1a78861680 x1874668973206656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.471843:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973206656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.471850:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.471854:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829836.471857:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973206656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.471863:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829836] req@ffff9c1a78862580 x1874668973206784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.471874:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973206784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.471881:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.471885:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829836.471888:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973206784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.471894:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829836] req@ffff9c1a788630c0 x1874668973206912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.471905:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973206912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.471913:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.471916:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829836.471919:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973206912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.471924:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829833/real 1787829836] req@ffff9c1a78860000 x1874668973206528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.471935:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973206528/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.471942:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.471945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829836.471949:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973206528:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829836.983423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829836.983423:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829837.239531:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829838.007500:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829838.007500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829838.263573:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829838.519531:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829838.519542:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829838.519547:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829838.519558:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6288 00000100:00080000:3.0:1787829838.519570:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829838.519578:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829838.519619:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973207424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829838.519664:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829838.519668:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829838.519670:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829838.519675:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6288 00000100:00080000:3.0:1787829838.519679:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829838.519683:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829838.519694:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829838.519707:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973207552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829838.519717:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829838.519721:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829838.519723:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829838.519726:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6288 00000100:00080000:3.0:1787829838.519730:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829838.519734:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829838.519738:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829838.519747:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973207680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829838.519757:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829838.519761:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829838.519763:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829838.519767:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6288 00000100:00080000:3.0:1787829838.519771:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829838.519775:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829838.519780:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829838.519789:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973207808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829838.519798:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829838.519801:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829838.519803:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829838.519807:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6288 00000100:00080000:3.0:1787829838.519810:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829838.519813:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829838.519819:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829838.519829:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973207936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829838.519838:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829838.519841:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829838.519843:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829838.519846:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6288 00000100:00080000:3.0:1787829838.519850:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829838.519854:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829838.519860:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829838.519869:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973208064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829838.519884:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829838.519888:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829838.519889:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829838.519893:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6288 00000100:00080000:3.0:1787829838.519896:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829838.519900:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829838.519903:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829838.519911:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973208192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829838.520018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973207424:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787829838.520102:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829838.520155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829838/real 1787829838] req@ffff9c1a7895d2c0 x1874668973207424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829838.520176:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973207424/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829838.520187:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829838.520191:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829838.520195:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973207424:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829838.520203:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973207552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829838.520225:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829838.520237:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829838/real 1787829838] req@ffff9c1a7895cb40 x1874668973207552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829838.520255:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973207552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829838.520263:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829838.520266:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829838.520270:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973207552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829838.520276:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973207680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829838.520293:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829838.520302:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829838/real 1787829838] req@ffff9c1a7895e1c0 x1874668973207680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829838.520313:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973207680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829838.520321:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829838.520325:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829838.520328:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973207680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829838.520334:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973207808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829838.520383:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829838.520395:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829838/real 1787829838] req@ffff9c1a7895da40 x1874668973207808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829838.520406:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973207808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829838.520422:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829838.520425:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829838.520428:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973207808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829838.520435:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973207936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829838.520451:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829838.520460:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829838/real 1787829838] req@ffff9c1a7895de00 x1874668973207936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829838.520471:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973207936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829838.520479:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829838.520482:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829838.520486:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973207936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829838.520490:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973208064:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829838.520508:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829838.520518:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829838/real 1787829838] req@ffff9c1a7895f840 x1874668973208064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829838.520530:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973208064/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829838.520538:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829838.520541:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829838.520544:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973208064:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829838.520550:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973208192:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829838.520568:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829838.520578:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829838/real 1787829838] req@ffff9c1a7895f0c0 x1874668973208192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829838.520589:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973208192/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829838.520598:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829838.520601:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829838.520604:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973208192:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829839.031424:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829839.031424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829839.287510:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829840.055433:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829840.055445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829840.311550:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829841.079449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829841.079457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829841.335537:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829842.103443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829842.103443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829842.359528:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829843.127411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829843.127439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829843.383462:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829843.639458:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829843.639465:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829843.639468:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829843.639475:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6293 00000100:00080000:0.0:1787829843.639482:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829843.639489:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829843.639526:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973208320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829843.639546:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829843.639548:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829843.639550:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829843.639552:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6293 00000100:00080000:0.0:1787829843.639555:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829843.639557:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829843.639565:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829843.639573:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973208448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829843.639580:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829843.639582:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829843.639584:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829843.639587:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973208320:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829843.639589:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6293 00000100:00080000:0.0:1787829843.639595:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829843.639599:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829843.639605:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829843.639615:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973208576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829843.639625:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829843.639629:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829843.639631:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829843.639635:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6293 00000100:00080000:0.0:1787829843.639639:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829843.639640:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829843.639644:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829843.639649:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829843.639662:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973208704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829843.639668:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829843.639671:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829843.639673:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829843.639674:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829843/real 1787829843] req@ffff9c1a78860780 x1874668973208320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829843.639678:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6293 00000100:00080000:0.0:1787829843.639681:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829843.639683:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829843.639685:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829843.639692:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973208832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829843.639700:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829843.639702:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829843.639703:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829843.639707:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6293 00000100:00100000:2.0:1787829843.639708:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973208320/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829843.639709:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829843.639713:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829843.639719:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829843.639727:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973208960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829843.639729:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829843.639734:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829843.639738:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829843.639739:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829843.639740:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829843.639745:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6293 00000100:00100000:2.0:1787829843.639748:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973208320:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829843.639749:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829843.639754:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829843.639759:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829843.639764:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973209088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829843.639788:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973208448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829843.639810:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829843.639819:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829843/real 1787829843] req@ffff9c1a78862940 x1874668973208448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829843.639827:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973208448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829843.639833:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829843.639835:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829843.639837:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973208448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829843.639840:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973208576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829843.639851:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829843.639857:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829843/real 1787829843] req@ffff9c1a78861a40 x1874668973208576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829843.639863:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973208576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829843.639867:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829843.639869:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829843.639871:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973208576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829843.639874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973208704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829843.639883:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829843.639887:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829843/real 1787829843] req@ffff9c1a78860f00 x1874668973208704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829843.639893:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973208704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829843.639899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829843.639900:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829843.639902:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973208704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829843.639905:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973208832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829843.639913:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829843.639918:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829843/real 1787829843] req@ffff9c1a788621c0 x1874668973208832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829843.639923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973208832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829843.639928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829843.639930:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829843.639931:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973208832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829843.639935:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973208960:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829843.639960:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829843.639967:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829843/real 1787829843] req@ffff9c1a78863840 x1874668973208960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829843.639974:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973208960/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829843.639978:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829843.639980:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829843.639982:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973208960:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829843.639986:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973209088:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829843.639995:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829843.640000:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829843/real 1787829843] req@ffff9c1a78862d00 x1874668973209088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829843.640006:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973209088/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829843.640010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829843.640012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829843.640013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973209088:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829844.151434:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829844.151445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829844.407523:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829845.175483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829845.175483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829845.431576:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829846.199427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829846.199427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829846.455541:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829847.223474:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829847.223481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829847.479592:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829848.247443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829848.247459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829848.503513:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829848.759480:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829848.759491:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829848.759495:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829848.759503:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6299 00000100:00080000:1.0:1787829848.759512:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829848.759518:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829848.759550:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973209216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829848.759588:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829848.759592:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829848.759594:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829848.759598:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6299 00000100:00080000:1.0:1787829848.759602:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829848.759607:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829848.759616:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829848.759633:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973209344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829848.759647:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829848.759651:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829848.759654:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829848.759663:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6299 00000100:00100000:2.0:1787829848.759663:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973209216:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829848.759672:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829848.759678:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829848.759683:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829848.759697:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973209472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829848.759707:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829848.759711:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829848.759714:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829848.759719:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6299 00000100:00080000:1.0:1787829848.759725:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829848.759729:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829848.759736:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829848.759750:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973209600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829848.759756:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829848.759771:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829848.759774:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829848.759776:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829848.759782:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6299 00000100:00080000:1.0:1787829848.759785:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829848.759788:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829848.759792:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829848.759803:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973209728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829848.759814:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829848.759817:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829848.759820:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829848.759825:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6299 00000100:00100000:2.0:1787829848.759829:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829848/real 1787829848] req@ffff9c1a78a792c0 x1874668973209216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829848.759833:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829848.759845:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829848.759861:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829848.759871:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973209216/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829848.759877:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973209856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829848.759897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829848.759918:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829848.759921:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829848.759922:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829848.759938:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787829848.759939:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6299 00000100:00080000:1.0:1787829848.759949:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829848.759952:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973209216:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829848.759959:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829848.759966:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829848.759978:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973209984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829848.759988:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973209344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829848.760015:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829848.760027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829848/real 1787829848] req@ffff9c1a78a78000 x1874668973209344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829848.760036:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973209344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829848.760045:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829848.760049:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829848.760053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973209344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829848.760061:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973209472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829848.760083:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829848.760096:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829848/real 1787829848] req@ffff9c1a78a79a40 x1874668973209472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829848.760108:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973209472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829848.760114:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829848.760116:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829848.760118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973209472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829848.760124:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973209600:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829848.760139:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829848.760147:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829848/real 1787829848] req@ffff9c1a78a7b480 x1874668973209600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829848.760155:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973209600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829848.760161:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829848.760163:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829848.760166:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973209600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829848.760169:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973209728:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829848.760179:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829848.760185:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829848/real 1787829848] req@ffff9c1a78a78f00 x1874668973209728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829848.760192:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973209728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829848.760197:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829848.760199:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829848.760201:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973209728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829848.760205:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973209856:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829848.760216:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829848.760221:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829848/real 1787829848] req@ffff9c1a78a7a580 x1874668973209856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829848.760228:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973209856/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829848.760233:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829848.760235:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829848.760238:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973209856:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829848.760254:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973209984:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829848.760265:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829848.760271:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829848/real 1787829848] req@ffff9c1a78a7b840 x1874668973209984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829848.760279:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973209984/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829848.760284:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829848.760286:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829848.760288:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973209984:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829849.271512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829849.271526:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829849.527569:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829850.295455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829850.295487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829850.551498:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829851.319503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829851.319628:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829851.575586:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829852.343477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829852.343481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829852.599611:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829853.367496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829853.367497:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829853.623542:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829853.879468:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829853.879479:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829853.879484:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829853.879496:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6304 00000100:00080000:2.0:1787829853.879507:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829853.879515:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829853.879549:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973210112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829853.879590:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829853.879596:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829853.879598:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829853.879604:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6304 00000100:00080000:2.0:1787829853.879610:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829853.879615:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829853.879625:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829853.879630:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973210112:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829853.879644:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973210240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829853.879658:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829853.879663:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829853.879665:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829853.879670:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6304 00000100:00080000:2.0:1787829853.879676:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829853.879681:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829853.879686:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829853.879701:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829853.879703:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973210368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829853.879718:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829853.879722:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829853.879724:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829853.879728:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6304 00000100:00080000:2.0:1787829853.879733:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829853.879736:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829853/real 1787829853] req@ffff9c1a49d8e1c0 x1874668973210112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829853.879741:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829853.879753:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829853.879759:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973210112/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829853.879768:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973210496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829853.879769:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829853.879778:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829853.879787:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973210112:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829853.879789:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829853.879794:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829853.879796:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829853.879802:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6304 00000100:00080000:2.0:1787829853.879807:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829853.879811:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973210240:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829853.879815:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829853.879822:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829853.879834:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829853.879838:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973210624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829853.879853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829853.879854:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829853/real 1787829853] req@ffff9c1a49d8fc00 x1874668973210240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829853.879861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829853.879866:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829853.879889:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6304 00000100:00080000:2.0:1787829853.879896:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829853.879901:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973210240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829853.879903:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787829853.879915:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829853.879915:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829853.879922:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829853.879925:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973210240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829853.879934:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973210368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829853.879935:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973210752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829853.879950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829853.879955:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787829853.879959:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829853.879960:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829853.879968:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6304 00000100:00100000:1.0:1787829853.879973:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829853/real 1787829853] req@ffff9c1a49d8ed00 x1874668973210368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829853.879974:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829853.879983:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829853.879990:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973210368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829853.879991:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829853.880004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829853.880012:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973210880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829853.880017:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829853.880024:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973210368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829853.880033:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973210496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829853.880052:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829853.880064:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829853/real 1787829853] req@ffff9c1a49d8f0c0 x1874668973210496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829853.880072:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973210496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829853.880077:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829853.880079:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829853.880081:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973210496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829853.880094:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973210624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829853.880104:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829853.880111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829853/real 1787829853] req@ffff9c1a49d8c3c0 x1874668973210624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829853.880118:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973210624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829853.880123:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829853.880125:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829853.880127:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973210624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829853.880130:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973210752:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829853.880148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829853.880154:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829853/real 1787829853] req@ffff9c1a49d8e580 x1874668973210752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829853.880162:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973210752/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829853.880167:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829853.880169:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829853.880171:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973210752:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829853.880175:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973210880:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829853.880184:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829853.880190:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829853/real 1787829853] req@ffff9c1a49d8d680 x1874668973210880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829853.880197:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973210880/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829853.880202:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829853.880203:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829853.880205:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973210880:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829854.391452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829854.391469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829854.647525:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829855.415448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829855.415448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829855.671517:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829856.439410:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829856.439428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829856.695521:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829857.463452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829857.463480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829857.719523:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829858.487447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829858.487459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829858.743515:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829858.999429:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829858.999437:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829858.999441:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829858.999448:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6309 00000100:00080000:3.0:1787829858.999462:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829858.999468:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829858.999502:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973211008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829858.999524:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829858.999527:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829858.999529:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829858.999534:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6309 00000100:00080000:3.0:1787829858.999538:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829858.999542:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829858.999551:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829858.999566:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973211136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829858.999571:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973211008:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829858.999577:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829858.999580:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829858.999582:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829858.999585:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6309 00000100:00080000:3.0:1787829858.999589:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829858.999592:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829858.999597:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829858.999606:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973211264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829858.999615:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829858.999618:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829858.999620:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829858.999623:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6309 00000100:00080000:3.0:1787829858.999625:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829858.999628:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829858.999629:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829858.999633:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829858.999642:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973211392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829858.999650:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829858.999653:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829858.999655:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829858.999658:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6309 00000100:00080000:3.0:1787829858.999661:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829858.999664:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829858/real 1787829858] req@ffff9c1a7895fc00 x1874668973211008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829858.999664:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829858.999669:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829858.999679:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973211520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829858.999687:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829858.999689:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973211008/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829858.999690:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829858.999692:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829858.999695:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6309 00000100:00080000:3.0:1787829858.999698:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829858.999700:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829858.999701:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787829858.999704:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787829858.999705:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829858.999708:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973211008:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829858.999713:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973211648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829858.999721:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829858.999724:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829858.999726:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829858.999729:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973211136:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829858.999729:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6309 00000100:00080000:3.0:1787829858.999732:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829858.999735:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829858.999739:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829858.999749:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973211776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829858.999750:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829858.999763:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829858/real 1787829858] req@ffff9c1a7895f0c0 x1874668973211136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829858.999775:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973211136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829858.999784:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829858.999787:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829858.999791:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973211136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829858.999798:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973211264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829858.999816:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829858.999828:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829858/real 1787829858] req@ffff9c1a7895f840 x1874668973211264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829858.999854:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973211264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829858.999870:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829858.999873:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829858.999877:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973211264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829858.999884:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973211392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829858.999901:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829858.999910:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829858/real 1787829858] req@ffff9c1a7895de00 x1874668973211392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829858.999917:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973211392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829858.999923:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829858.999925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829858.999927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973211392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829858.999930:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973211520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829858.999940:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829858.999946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829858/real 1787829858] req@ffff9c1a7895da40 x1874668973211520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829858.999953:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973211520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829858.999958:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829858.999960:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829858.999962:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973211520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829858.999966:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973211648:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829858.999976:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829858.999982:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829858/real 1787829858] req@ffff9c1a7895e1c0 x1874668973211648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829858.999989:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973211648/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829858.999994:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829858.999996:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829858.999998:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973211648:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829859.000010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973211776:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829859.000020:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829859.000026:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829858/real 1787829859] req@ffff9c1a7895cb40 x1874668973211776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829859.000038:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973211776/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829859.000046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829859.000050:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829859.000053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973211776:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829859.511482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829859.511482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829859.767583:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829860.535486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829860.535486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829860.791549:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829861.559457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829861.559459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829861.815550:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829862.583455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829862.583456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829862.839513:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829863.607453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829863.607456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829863.863513:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829864.119481:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829864.119492:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829864.119496:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829864.119507:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6314 00000100:00080000:2.0:1787829864.119518:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829864.119525:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829864.119566:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973211904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829864.119593:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829864.119597:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829864.119599:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829864.119604:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6314 00000100:00080000:2.0:1787829864.119608:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829864.119612:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829864.119623:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829864.119638:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973212032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829864.119646:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973211904:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829864.119664:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829864.119669:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829864.119672:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829864.119677:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6314 00000100:00080000:2.0:1787829864.119683:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829864.119687:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829864.119693:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829864.119710:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829864.119711:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973212160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829864.119727:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829864.119731:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829864.119733:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829864.119738:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6314 00000100:00080000:2.0:1787829864.119743:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829864.119747:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829864.119752:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff9c1a49d8d680 x1874668973211904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829864.119755:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829864.119776:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973212288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829864.119778:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973211904/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829864.119818:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829864.119823:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829864.119826:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829864.119833:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829864.119834:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6314 00000100:00080000:2.0:1787829864.119843:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829864.119844:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829864.119853:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829864.119854:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973211904:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787829864.119862:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829864.119878:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973212416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829864.119885:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973212032:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829864.119897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829864.119902:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829864.119904:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829864.119910:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6314 00000100:00080000:2.0:1787829864.119916:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829864.119918:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829864.119925:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829864.119932:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829864.119937:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff9c1a49d8e580 x1874668973212032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829864.119949:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973212544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829864.119960:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973212032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829864.119963:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829864.119974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829864.119979:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829864.119981:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829864.119986:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6314 00000100:00080000:1.0:1787829864.119989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829864.119994:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829864.119996:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973212032:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829864.120003:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829864.120008:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973212160:192.168.203.148@tcp:38:kworker.0 00000008:00100000:2.0:1787829864.120011:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829864.120029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973212672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829864.120035:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829864.120049:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff9c1a49d8c3c0 x1874668973212160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829864.120063:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973212160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829864.120071:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829864.120075:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829864.120079:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973212160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829864.120084:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973212288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829864.120100:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829864.120109:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff9c1a49d8f0c0 x1874668973212288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829864.120120:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973212288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829864.120128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829864.120131:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829864.120134:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973212288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829864.120149:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973212416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829864.120165:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829864.120174:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff9c1a49d8ed00 x1874668973212416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829864.120186:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973212416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829864.120194:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829864.120197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829864.120200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973212416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829864.120205:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973212544:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829864.120220:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829864.120229:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff9c1a49d8fc00 x1874668973212544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829864.120239:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973212544/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829864.120247:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829864.120250:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829864.120253:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973212544:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829864.120258:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973212672:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829864.120272:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829864.120280:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829864/real 1787829864] req@ffff9c1a49d8e1c0 x1874668973212672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829864.120291:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973212672/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829864.120304:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829864.120307:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829864.120310:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973212672:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829864.631452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829864.631452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829864.887519:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829865.655409:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829865.655456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829865.911542:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829866.679444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829866.679444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829866.935532:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829867.703515:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829867.703515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829867.959567:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829868.727480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829868.727480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829868.983572:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829869.239496:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829869.239507:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829869.239512:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829869.239523:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6319 00000100:00080000:3.0:1787829869.239536:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829869.239543:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829869.239588:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973212800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829869.239615:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829869.239618:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829869.239619:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829869.239623:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6319 00000100:00080000:3.0:1787829869.239626:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829869.239628:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829869.239637:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829869.239647:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973212928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829869.239660:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829869.239663:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829869.239664:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829869.239669:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6319 00000100:00100000:1.0:1787829869.239670:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973212800:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829869.239671:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829869.239683:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829869.239689:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829869.239699:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973213056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829869.239706:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829869.239710:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829869.239711:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829869.239714:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6319 00000100:00080000:3.0:1787829869.239716:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829869.239719:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829869.239722:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829869.239728:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973213184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829869.239741:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829869.239744:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829869.239747:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829869.239754:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6319 00000800:00000100:1.0:1787829869.239758:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829869.239774:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829869.239782:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829869.239788:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829869.239804:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973213312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829869.239816:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829869.239818:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff9c1a7895d2c0 x1874668973212800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829869.239820:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829869.239824:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829869.239833:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6319 00000100:00080000:3.0:1787829869.239838:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829869.239841:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829869.239845:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973212800/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829869.239846:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829869.239860:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829869.239862:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973213440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829869.239866:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829869.239874:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973212800:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829869.239877:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829869.239882:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829869.239884:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829869.239887:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6319 00000100:00080000:3.0:1787829869.239889:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829869.239892:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829869.239894:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829869.239902:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973213568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829869.239906:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973212928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829869.239935:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829869.239946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff9c1a7895f480 x1874668973212928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.239956:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973212928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.239972:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.239975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829869.239977:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973212928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829869.239982:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973213056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829869.239995:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829869.240001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff9c1a7895c000 x1874668973213056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.240009:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973213056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.240015:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.240017:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829869.240019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973213056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829869.240023:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973213184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829869.240033:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829869.240039:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff9c1a7895e580 x1874668973213184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.240046:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973213184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.240057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.240059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829869.240061:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973213184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829869.240064:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973213312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829869.240074:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829869.240080:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff9c1a7895cf00 x1874668973213312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.240086:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973213312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.240092:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.240094:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829869.240096:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973213312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829869.240099:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973213440:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829869.240117:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829869.240128:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff9c1a7895e940 x1874668973213440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.240141:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973213440/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.240151:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.240154:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829869.240159:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973213440:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829869.240180:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973213568:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829869.240212:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829869.240223:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829869/real 1787829869] req@ffff9c1a7895c780 x1874668973213568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829869.240232:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973213568/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829869.240238:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829869.240240:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829869.240242:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973213568:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829869.751471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829869.751471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829870.007479:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829870.775461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829870.775558:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829871.031582:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829871.799449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829871.799452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829872.055540:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829872.823473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829872.823473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829873.079577:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829873.847475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829873.847487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829874.103629:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829874.359479:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829874.359491:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829874.359496:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829874.359506:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6324 00000100:00080000:3.0:1787829874.359516:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829874.359522:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829874.359560:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973213696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829874.359601:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829874.359606:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829874.359608:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829874.359613:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6324 00000100:00080000:3.0:1787829874.359617:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829874.359622:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829874.359633:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829874.359650:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973213696:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829874.359653:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973213824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829874.359669:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829874.359673:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829874.359675:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829874.359679:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6324 00000100:00080000:3.0:1787829874.359684:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829874.359688:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829874.359694:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829874.359710:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973213952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829874.359719:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829874.359742:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829874.359748:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829874.359750:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829874.359755:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6324 00000100:00080000:3.0:1787829874.359761:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829874.359766:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829874.359771:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff9c1a7895c3c0 x1874668973213696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829874.359774:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829874.359793:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973214080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829874.359796:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973213696/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829874.359821:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829874.359826:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829874.359829:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829874.359831:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829874.359835:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6324 00000100:00080000:1.0:1787829874.359841:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829874.359843:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829874.359851:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973213696:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829874.359852:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829874.359862:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829874.359878:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973214208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829874.359883:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973213824:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829874.359900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829874.359905:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829874.359908:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829874.359915:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6324 00000800:00000100:1.0:1787829874.359916:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829874.359923:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829874.359931:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829874.359934:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff9c1a7895c780 x1874668973213824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829874.359940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829874.359949:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973213824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829874.359957:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829874.359959:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973214336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829874.359962:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829874.359969:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973213824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829874.359980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973213952:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829874.359988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829874.359993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829874.359996:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829874.360000:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829874.360005:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6324 00000100:00100000:1.0:1787829874.360011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff9c1a7895e940 x1874668973213952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829874.360013:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829874.360021:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829874.360026:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829874.360039:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973213952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829874.360048:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973214464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829874.360050:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829874.360057:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829874.360065:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973213952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829874.360077:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973214080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829874.360099:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829874.360107:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff9c1a7895cf00 x1874668973214080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829874.360116:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973214080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829874.360122:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829874.360124:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829874.360126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973214080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829874.360137:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973214208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829874.360148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829874.360154:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff9c1a7895e580 x1874668973214208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829874.360161:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973214208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829874.360166:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829874.360168:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829874.360178:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973214208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829874.360182:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973214336:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829874.360193:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829874.360200:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff9c1a7895c000 x1874668973214336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829874.360213:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973214336/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829874.360218:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829874.360220:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829874.360222:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973214336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829874.360225:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973214464:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829874.360234:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829874.360240:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829874/real 1787829874] req@ffff9c1a7895f480 x1874668973214464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829874.360247:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973214464/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829874.360252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829874.360254:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829874.360256:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973214464:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829874.871457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829874.871457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829875.127607:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829875.895454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829875.895491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829876.151568:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829876.919489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829876.919497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829877.175597:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829877.943460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829877.943479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829878.199598:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829878.967455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829878.967456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829879.223589:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829879.479459:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829879.479467:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829879.479471:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829879.479480:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6329 00000100:00080000:3.0:1787829879.479489:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829879.479495:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829879.479527:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973214592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829879.479552:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829879.479554:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829879.479556:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829879.479559:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6329 00000100:00080000:3.0:1787829879.479561:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829879.479563:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829879.479570:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829879.479578:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973214720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829879.479589:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829879.479591:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829879.479593:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829879.479595:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6329 00000100:00080000:3.0:1787829879.479596:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829879.479598:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829879.479601:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829879.479606:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973214848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829879.479613:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829879.479616:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829879.479617:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829879.479620:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6329 00000100:00080000:3.0:1787829879.479622:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829879.479624:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829879.479627:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829879.479635:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973214592:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829879.479636:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973214976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829879.479655:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829879.479656:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829879.479658:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829879.479660:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6329 00000100:00080000:3.0:1787829879.479664:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829879.479666:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829879.479668:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829879.479679:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973215104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829879.479688:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829879.479705:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829879.479707:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829879.479711:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6329 00000100:00080000:3.0:1787829879.479715:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829879.479719:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829879.479724:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829879.479731:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829879.479743:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973215232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829879.479755:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829879.479756:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829879/real 1787829879] req@ffff9c1a7895d2c0 x1874668973214592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829879.479761:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829879.479766:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829879.479774:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6329 00000100:00080000:3.0:1787829879.479778:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829879.479780:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973214592/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829879.479788:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787829879.479794:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829879.479796:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829879.479799:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829879.479804:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973214592:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829879.479813:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973214720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829879.479813:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973215360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829879.479838:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829879.479849:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829879/real 1787829879] req@ffff9c1a7895cb40 x1874668973214720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829879.479859:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973214720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829879.479865:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829879.479867:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829879.479869:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973214720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829879.479873:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973214848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829879.479883:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829879.479888:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829879/real 1787829879] req@ffff9c1a7895e1c0 x1874668973214848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829879.479900:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973214848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829879.479905:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829879.479906:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829879.479908:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973214848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829879.479927:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973214976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829879.479937:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829879.479943:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829879/real 1787829879] req@ffff9c1a7895da40 x1874668973214976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829879.479951:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973214976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829879.479956:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829879.479958:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829879.479959:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973214976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829879.479963:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973215104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829879.479972:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829879.479977:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829879/real 1787829879] req@ffff9c1a7895de00 x1874668973215104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829879.479984:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973215104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829879.479993:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829879.479995:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829879.479997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973215104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829879.480000:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973215232:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829879.480010:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829879.480015:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829879/real 1787829879] req@ffff9c1a7895f840 x1874668973215232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829879.480022:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973215232/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829879.480027:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829879.480029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829879.480031:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973215232:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829879.480034:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973215360:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829879.480043:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829879.480048:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829879/real 1787829879] req@ffff9c1a7895f0c0 x1874668973215360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829879.480054:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973215360/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829879.480059:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829879.480061:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829879.480063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973215360:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829879.991449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829879.991476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829880.247519:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829881.015460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829881.015460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829881.271507:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829882.039462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829882.039462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829882.295479:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829883.063431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829883.063470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829883.319543:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829884.087450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829884.087450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829884.343670:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829884.599480:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829884.599491:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829884.599496:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829884.599505:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6334 00000100:00080000:2.0:1787829884.599520:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829884.599527:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829884.599565:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973215488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829884.599592:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829884.599596:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829884.599598:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829884.599602:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6334 00000100:00080000:2.0:1787829884.599606:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829884.599610:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829884.599619:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829884.599633:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973215616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829884.599644:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829884.599651:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973215488:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829884.599651:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829884.599654:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829884.599674:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6334 00000100:00080000:2.0:1787829884.599678:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829884.599682:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829884.599688:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829884.599702:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973215744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829884.599713:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829884.599717:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829884.599719:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829884.599723:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6334 00000100:00080000:2.0:1787829884.599726:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829884.599732:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829884.599733:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829884.599741:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829884.599754:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973215872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829884.599763:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829884.599766:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829884.599768:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829884.599772:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6334 00000100:00080000:2.0:1787829884.599777:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829884.599782:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829884/real 1787829884] req@ffff9c1a49d8cb40 x1874668973215488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829884.599783:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829884.599791:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829884.599808:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973216000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829884.599815:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973215488/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829884.599823:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829884.599832:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829884.599835:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829884.599836:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829884.599843:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787829884.599851:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6334 00000100:00100000:1.0:1787829884.599853:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973215488:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829884.599857:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829884.599865:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829884.599870:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829884.599883:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973215616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829884.599885:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973216128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829884.599900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829884.599903:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829884.599905:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829884.599909:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6334 00000100:00080000:2.0:1787829884.599913:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829884.599918:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829884.599923:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829884.599926:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829884.599940:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973216256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829884.599946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829884/real 1787829884] req@ffff9c1a49d8cf00 x1874668973215616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.599968:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973215616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.599985:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.599989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829884.599993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973215616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829884.599999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973215744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829884.600020:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829884.600033:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829884/real 1787829884] req@ffff9c1a49d8c000 x1874668973215744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.600047:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973215744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.600056:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.600060:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829884.600063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973215744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829884.600070:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973215872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829884.600091:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829884.600103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829884/real 1787829884] req@ffff9c1a49d8c780 x1874668973215872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.600116:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973215872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.600125:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.600129:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829884.600132:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973215872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829884.600139:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973216000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829884.600161:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829884.600173:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829884/real 1787829884] req@ffff9c1a49d8de00 x1874668973216000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.600185:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973216000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.600193:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.600197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829884.600200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973216000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829884.600220:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973216128:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829884.600242:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829884.600253:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829884/real 1787829884] req@ffff9c1a49d8da40 x1874668973216128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.600266:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973216128/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.600274:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.600277:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829884.600281:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973216128:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829884.600293:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973216256:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829884.600313:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829884.600326:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829884/real 1787829884] req@ffff9c1a49d8cb40 x1874668973216256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829884.600338:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973216256/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829884.600376:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829884.600380:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829884.600384:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973216256:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829885.111439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829885.111446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829885.367532:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829886.135521:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829886.135522:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829886.391612:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829887.159439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829887.159446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829887.415593:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829888.183444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829888.183451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829888.439585:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829889.207449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829889.207449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829889.463519:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829889.719469:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829889.719477:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829889.719481:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829889.719489:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6340 00000100:00080000:3.0:1787829889.719498:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829889.719505:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829889.719539:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973216384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829889.719568:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829889.719571:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829889.719573:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829889.719578:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6340 00000100:00080000:3.0:1787829889.719581:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829889.719585:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829889.719595:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829889.719610:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973216512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829889.719632:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829889.719636:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829889.719637:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973216384:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829889.719641:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829889.719657:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6340 00000100:00080000:3.0:1787829889.719663:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829889.719669:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829889.719676:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829889.719694:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973216640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829889.719707:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829889.719711:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829889.719714:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829889.719719:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6340 00000100:00080000:3.0:1787829889.719725:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829889.719730:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829889.719731:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829889.719744:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829889.719759:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973216768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829889.719772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829889.719775:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829889.719777:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829889.719781:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6340 00000100:00080000:3.0:1787829889.719785:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829889.719789:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829889/real 1787829889] req@ffff9c1a7895fc00 x1874668973216384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829889.719794:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829889.719802:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829889.719821:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973216896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829889.719828:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973216384/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829889.719840:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829889.719845:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829889.719848:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973216384:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829889.719874:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973216512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829889.719895:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829889.719907:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829889.719910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829889.719912:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829889.719917:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829889/real 1787829889] req@ffff9c1a7895f0c0 x1874668973216512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829889.719917:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6340 00000100:00080000:3.0:1787829889.719931:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829889.719943:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829889.719947:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829889.719955:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973216512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829889.719962:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973217024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829889.719999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829889.720002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829889.720003:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829889.720004:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829889.720019:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829889.720020:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6340 00000100:00080000:3.0:1787829889.720028:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829889.720032:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973216512:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829889.720039:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829889.720047:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829889.720055:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973216640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829889.720084:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829889.720094:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829889/real 1787829889] req@ffff9c1a7895f840 x1874668973216640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829889.720110:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973216640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829889.720116:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829889.720118:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829889.720121:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973216640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829889.720126:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973216768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829889.720141:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829889.720155:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829889/real 1787829889] req@ffff9c1a7895de00 x1874668973216768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829889.720166:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973216768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829889.720172:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829889.720175:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829889.720177:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973216768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829889.720197:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973217152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829889.720200:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973216896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829889.720231:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829889.720241:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829889/real 1787829889] req@ffff9c1a7895da40 x1874668973216896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829889.720259:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973216896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829889.720266:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829889.720271:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829889.720275:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973216896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829889.720281:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973217024:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829889.720313:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829889.720336:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829889/real 1787829889] req@ffff9c1a7895e1c0 x1874668973217024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829889.720382:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973217024/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829889.720388:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829889.720391:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829889.720393:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973217024:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829889.720418:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973217152:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829889.720438:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829889.720446:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829889/real 1787829889] req@ffff9c1a7895cb40 x1874668973217152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829889.720454:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973217152/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829889.720466:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829889.720468:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829889.720470:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973217152:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829890.231512:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829890.231512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829890.487567:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829891.255458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829891.255460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829891.511559:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829892.279465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829892.279472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829892.535586:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829893.303428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829893.303435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829893.559583:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829894.327489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829894.327508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829894.583555:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829894.839507:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829894.839520:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829894.839526:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829894.839539:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6345 00000100:00080000:2.0:1787829894.839553:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829894.839561:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829894.839628:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973217280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829894.839661:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829894.839664:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829894.839666:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829894.839669:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6345 00000100:00080000:2.0:1787829894.839673:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829894.839676:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829894.839687:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829894.839698:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973217408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829894.839707:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829894.839709:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829894.839710:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829894.839712:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6345 00000100:00080000:2.0:1787829894.839715:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829894.839720:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829894.839721:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973217280:192.168.203.148@tcp:250:kworker.0 00000008:00100000:2.0:1787829894.839724:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829894.839735:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973217536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829894.839743:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829894.839745:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829894.839746:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829894.839749:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6345 00000100:00080000:2.0:1787829894.839752:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829894.839754:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829894.839757:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829894.839766:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973217664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829894.839773:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829894.839775:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829894.839776:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829894.839779:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6345 00000100:00080000:2.0:1787829894.839782:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829894.839784:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829894.839787:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829894.839794:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973217792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829894.839799:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829894.839803:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787829894.839805:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829894.839806:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829894.839812:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6345 00000100:00080000:2.0:1787829894.839815:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829894.839819:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829894.839822:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829894.839831:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973217920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829894.839843:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829894.839845:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829894.839847:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829894.839850:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6345 00000100:00080000:2.0:1787829894.839852:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829894.839854:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829894/real 1787829894] req@ffff9c1a49d8e1c0 x1874668973217280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829894.839858:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829894.839865:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829894.839876:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973218048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829894.839881:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973217280/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829894.839905:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829894.839912:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829894.839918:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973217280:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829894.839946:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973217408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829894.839967:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829894.839977:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829894/real 1787829894] req@ffff9c1a49d8fc00 x1874668973217408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829894.839986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973217408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829894.839992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829894.839994:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829894.839997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973217408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829894.840001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973217536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829894.840013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829894.840019:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829894/real 1787829894] req@ffff9c1a49d8ed00 x1874668973217536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829894.840027:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973217536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829894.840035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829894.840038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829894.840042:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973217536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829894.840049:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973217664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829894.840069:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829894.840082:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829894/real 1787829894] req@ffff9c1a49d8f0c0 x1874668973217664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829894.840093:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973217664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829894.840099:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829894.840101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829894.840104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973217664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829894.840108:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973217792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829894.840123:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829894.840130:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829894/real 1787829894] req@ffff9c1a49d8c3c0 x1874668973217792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829894.840138:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973217792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829894.840144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829894.840146:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829894.840148:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973217792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829894.840151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973217920:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829894.840162:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829894.840175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829894/real 1787829894] req@ffff9c1a49d8e580 x1874668973217920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829894.840184:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973217920/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829894.840190:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829894.840193:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829894.840196:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973217920:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829894.840202:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973218048:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829894.840221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829894.840232:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829894/real 1787829894] req@ffff9c1a49d8d680 x1874668973218048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829894.840246:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973218048/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829894.840253:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829894.840255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829894.840258:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973218048:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829895.351439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829895.351461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829895.607551:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829896.375462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829896.375462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829896.631546:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829897.399417:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829897.399437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829897.655613:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829898.423505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829898.425723:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829898.679619:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829899.447479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829899.447480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829899.703521:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829899.959478:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829899.959487:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829899.959491:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829899.959501:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6350 00000100:00080000:0.0:1787829899.959510:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829899.959517:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829899.959559:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973218176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829899.959589:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829899.959593:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829899.959595:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829899.959598:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6350 00000100:00080000:0.0:1787829899.959602:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829899.959606:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829899.959615:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829899.959623:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973218176:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829899.959630:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973218304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829899.959641:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829899.959645:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829899.959647:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829899.959651:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6350 00000100:00080000:0.0:1787829899.959654:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829899.959657:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829899.959661:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829899.959670:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973218432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829899.959680:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829899.959683:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829899.959685:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829899.959688:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6350 00000100:00080000:0.0:1787829899.959692:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829899.959695:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829899.959699:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829899.959708:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973218560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829899.959718:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829899.959721:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829899.959724:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829899.959729:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6350 00000100:00100000:1.0:1787829899.959729:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973218304:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829899.959737:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829899.959742:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829899.959747:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829899.959758:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973218688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829899.959761:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973218432:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829899.959775:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829899.959778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829899.959780:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829899.959782:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973218560:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829899.959787:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6350 00000100:00080000:0.0:1787829899.959793:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829899.959797:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829899.959802:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829899.959812:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973218688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829899.959815:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973218816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829899.959832:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829899.959834:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973218816:192.168.203.148@tcp:8:kworker.0 00000100:00080000:0.0:1787829899.959839:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829899.959842:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829899.959847:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6350 00000100:00080000:0.0:1787829899.959852:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829899.959856:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829899.959861:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829899.959871:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973218944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829899.959908:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973218944:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829900.471430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829900.471431:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829900.727530:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829901.495446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829901.495446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829901.751508:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829902.519439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829902.519458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829902.775483:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829903.031688:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:2.0:1787829903.031737:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829903.031755:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829903.031796:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829903.031807:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829903.031818:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829903.031829:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829903.031839:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:2.0:1787829903.031849:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787829903.031851:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829899/real 1787829903] req@ffff9c1a78860f00 x1874668973218688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829903.031883:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973218688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829903.031899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829903.031905:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829903.031910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973218688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829903.031918:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829899/real 1787829903] req@ffff9c1a78862d00 x1874668973218304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829903.031933:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973218304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829903.031943:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829903.031947:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829903.031951:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973218304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829903.031957:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829899/real 1787829903] req@ffff9c1a78863840 x1874668973218432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829903.031970:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973218432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829903.031978:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829903.031981:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829903.031985:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973218432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829903.031992:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829899/real 1787829903] req@ffff9c1a788621c0 x1874668973218560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829903.032003:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973218560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829903.032011:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829903.032015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829903.032018:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973218560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829903.032025:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829899/real 1787829903] req@ffff9c1a78860b40 x1874668973218176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829903.032038:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973218176/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829903.032047:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829903.032051:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829903.032054:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973218176:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829903.032085:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829899/real 1787829903] req@ffff9c1a78862940 x1874668973218944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829903.032098:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973218944/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829903.032106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829903.032110:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829903.032113:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973218944:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829903.032121:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829899/real 1787829903] req@ffff9c1a78861a40 x1874668973218816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829903.032132:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973218816/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829903.032140:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829903.032143:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829903.032147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973218816:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829903.543442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829903.543442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829903.799533:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829904.567452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829904.567453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829904.823566:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829905.079475:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.079485:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829905.079489:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.079500:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6355 00000100:00080000:0.0:1787829905.079526:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.079534:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829905.079583:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973219072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829905.079611:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.079615:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829905.079617:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.079621:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6355 00000100:00080000:0.0:1787829905.079624:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.079629:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829905.079638:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829905.079652:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973219200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829905.079663:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973219072:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829905.079665:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.079673:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829905.079675:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.079679:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6355 00000100:00080000:0.0:1787829905.079683:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.079686:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829905.079691:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829905.079702:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973219328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829905.079711:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.079716:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829905.079717:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.079721:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6355 00000100:00080000:0.0:1787829905.079725:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.079729:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829905.079733:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829905.079745:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973219456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829905.079755:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.079758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829905.079760:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.079766:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6355 00000100:00080000:0.0:1787829905.079770:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829905.079773:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829905.079777:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829905.079784:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829905.079796:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973219584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829905.079811:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.079816:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787829905.079817:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829905] req@ffff9c1a78860780 x1874668973219072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829905.079819:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.079829:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6355 00000100:00080000:0.0:1787829905.079836:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.079842:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829905.079847:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973219072/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829905.079848:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829905.079863:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829905.079865:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973219712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829905.079872:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829905.079880:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973219072:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829905.079881:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829905.079889:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829905.079891:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829905.079896:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6355 00000100:00080000:0.0:1787829905.079899:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829905.079904:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829905.079908:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829905.079919:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973219840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829905.079924:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973219200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829905.079962:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829905.079976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829905] req@ffff9c1a78860000 x1874668973219200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829905.079988:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973219200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829905.079995:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829905.079998:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829905.080004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973219200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829905.080010:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973219328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829905.080028:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829905.080051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829905] req@ffff9c1a788630c0 x1874668973219328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829905.080069:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973219328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829905.080075:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829905.080078:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829905.080081:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973219328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829905.080086:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973219456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829905.080109:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829905.080120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829905] req@ffff9c1a78862580 x1874668973219456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829905.080131:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973219456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829905.080139:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829905.080142:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829905.080146:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973219456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829905.080151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973219584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829905.080169:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829905.080178:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829905] req@ffff9c1a78861680 x1874668973219584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829905.080188:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973219584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829905.080200:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829905.080202:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829905.080205:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973219584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829905.080210:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973219712:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829905.080226:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829905.080237:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829905] req@ffff9c1a78861e00 x1874668973219712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829905.080246:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973219712/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829905.080253:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829905.080256:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829905.080258:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973219712:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829905.080276:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973219840:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829905.080301:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829905.080311:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829905/real 1787829905] req@ffff9c1a788612c0 x1874668973219840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829905.080321:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973219840/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829905.080328:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829905.080331:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829905.080334:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973219840:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829905.591448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829905.591448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829905.847554:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829906.615452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829906.615452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829906.871477:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829907.639436:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829907.639459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829907.895581:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829908.663482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829908.663482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829908.919524:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829909.687453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829909.687453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829909.943541:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829910.199524:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829910.199536:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829910.199541:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829910.199554:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6360 00000100:00080000:3.0:1787829910.199567:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829910.199575:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829910.199627:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973219968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829910.199657:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829910.199661:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829910.199663:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829910.199667:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6360 00000100:00080000:3.0:1787829910.199671:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829910.199676:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829910.199686:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829910.199702:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973219968:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829910.199703:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973220096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829910.199718:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829910.199721:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829910.199724:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829910.199728:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6360 00000100:00080000:3.0:1787829910.199732:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829910.199736:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829910.199741:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829910.199751:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973220224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829910.199762:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829910.199765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829910.199767:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829910.199770:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6360 00000100:00080000:3.0:1787829910.199774:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829910.199778:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829910.199782:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829910.199791:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973220352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829910.199800:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829910.199805:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829910.199806:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829910.199810:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829910.199814:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6360 00000100:00080000:3.0:1787829910.199818:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829910.199821:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829910.199826:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829910.199838:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973220480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829910.199847:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829910.199850:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829910.199852:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829910.199856:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6360 00000100:00080000:3.0:1787829910.199860:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829910.199864:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff9c1a7895d2c0 x1874668973219968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829910.199868:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829910.199876:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829910.199889:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973220608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829910.199894:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973219968/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829910.199904:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829910.199912:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829910.199914:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829910.199918:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829910.199919:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6360 00000100:00080000:3.0:1787829910.199924:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829910.199926:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829910.199931:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829910.199935:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973219968:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787829910.199938:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829910.199951:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973220736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829910.199970:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973220096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829910.199998:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829910.200011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff9c1a7895f480 x1874668973220096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.200023:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973220096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.200031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.200035:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829910.200039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973220096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829910.200045:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973220224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829910.200064:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829910.200074:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff9c1a7895c000 x1874668973220224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.200085:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973220224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.200092:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.200095:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829910.200098:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973220224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829910.200103:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973220352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829910.200119:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829910.200128:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff9c1a7895e580 x1874668973220352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.200138:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973220352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.200146:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.200149:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829910.200152:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973220352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829910.200165:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973220480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829910.200181:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829910.200190:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff9c1a7895cf00 x1874668973220480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.200201:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973220480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.200209:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.200211:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829910.200214:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973220480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829910.200219:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973220608:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829910.200235:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829910.200245:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff9c1a7895e940 x1874668973220608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.200256:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973220608/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.200264:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.200266:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829910.200269:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973220608:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829910.200281:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973220736:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829910.200297:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829910.200306:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829910/real 1787829910] req@ffff9c1a7895c780 x1874668973220736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829910.200322:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973220736/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829910.200329:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829910.200332:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829910.200335:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973220736:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829910.711456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829910.711456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829910.967505:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829911.735459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829911.735459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829911.991528:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829912.759495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829912.759498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829913.015614:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829913.783446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829913.783446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829914.039542:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829914.807415:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829914.807420:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829915.063526:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829915.319463:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829915.319473:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829915.319477:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829915.319496:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6365 00000100:00080000:1.0:1787829915.319507:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829915.319515:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829915.319549:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973220864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829915.319578:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829915.319582:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829915.319584:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829915.319588:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6365 00000100:00080000:1.0:1787829915.319592:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829915.319596:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829915.319605:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829915.319619:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973220992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829915.319630:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973220864:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829915.319632:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829915.319653:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829915.319655:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829915.319661:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6365 00000100:00080000:1.0:1787829915.319666:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829915.319669:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829915.319674:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829915.319687:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973221120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829915.319688:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829915.319702:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829915.319705:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829915.319708:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829915.319713:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6365 00000100:00080000:1.0:1787829915.319717:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829915.319721:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829915.319724:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff9c1a78a7b840 x1874668973220864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829915.319728:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829915.319741:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973220864/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829915.319746:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973221248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829915.319752:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829915.319758:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829915.319764:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973220864:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829915.319777:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829915.319780:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829915.319782:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829915.319787:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973220992:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829915.319788:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6365 00000100:00080000:1.0:1787829915.319796:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829915.319801:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829915.319806:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829915.319807:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829915.319818:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff9c1a78a7a580 x1874668973220992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829915.319823:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973221376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829915.319830:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973220992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829915.319841:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829915.319842:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829915.319846:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787829915.319850:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829915.319851:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973220992:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829915.319856:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829915.319859:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973221120:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829915.319864:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6365 00000100:00080000:1.0:1787829915.319870:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829915.319875:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829915.319876:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829915.319884:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829915.319891:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff9c1a78a78f00 x1874668973221120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829915.319899:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973221504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829915.319910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973221120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829915.319912:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829915.319921:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829915.319925:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829915.319928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829915.319932:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6365 00000100:00080000:2.0:1787829915.319936:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787829915.319939:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829915.319942:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973221120:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829915.319947:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829915.319950:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973221248:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787829915.319955:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829915.319972:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973221632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829915.319972:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829915.319982:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff9c1a78a7b480 x1874668973221248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829915.319998:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973221248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829915.320003:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829915.320006:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829915.320008:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973221248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829915.320018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973221376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829915.320029:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829915.320035:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff9c1a78a79a40 x1874668973221376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829915.320041:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973221376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829915.320046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829915.320048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829915.320049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973221376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829915.320052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973221504:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829915.320061:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829915.320066:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff9c1a78a78000 x1874668973221504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829915.320073:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973221504/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829915.320077:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829915.320079:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829915.320080:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973221504:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829915.320083:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973221632:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829915.320091:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829915.320095:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829915/real 1787829915] req@ffff9c1a78a792c0 x1874668973221632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829915.320101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973221632/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829915.320106:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829915.320107:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829915.320109:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973221632:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829915.831462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829915.831462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829916.087515:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829916.855414:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829916.855419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829917.111503:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829917.879466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829917.879466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829918.135481:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829918.903396:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829918.903407:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829919.159547:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829919.927469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829919.927469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829920.183556:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829920.439509:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829920.439522:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829920.439527:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829920.439538:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6370 00000100:00080000:1.0:1787829920.439551:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829920.439560:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829920.439607:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973221760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829920.439646:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829920.439651:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829920.439653:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829920.439658:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6370 00000100:00080000:1.0:1787829920.439661:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829920.439667:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829920.439678:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829920.439694:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973221888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829920.439694:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973221760:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829920.439710:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829920.439714:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829920.439716:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829920.439720:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6370 00000100:00080000:1.0:1787829920.439724:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829920.439728:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829920.439732:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829920.439743:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973222016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829920.439755:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829920.439758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829920.439760:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829920.439763:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6370 00000100:00080000:1.0:1787829920.439768:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829920.439772:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829920.439777:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829920.439782:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829920.439791:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973222144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829920.439801:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829920.439813:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829920.439814:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829920.439818:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6370 00000100:00080000:1.0:1787829920.439822:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829920.439827:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829920.439831:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff9c1a78a7bc00 x1874668973221760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829920.439834:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829920.439852:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973222272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829920.439856:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973221760/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829920.439875:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829920.439883:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829920.439887:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829920.439892:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829920.439893:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6370 00000100:00080000:1.0:1787829920.439901:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829920.439901:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829920.439907:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973221760:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787829920.439910:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829920.439918:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829920.439932:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973222400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829920.439936:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973221888:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787829920.439950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829920.439954:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829920.439957:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829920.439961:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6370 00000800:00000100:2.0:1787829920.439966:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787829920.439968:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829920.439975:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787829920.439982:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829920.439982:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff9c1a78a78780 x1874668973221888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829920.439999:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973221888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829920.440006:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973222528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829920.440010:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829920.440019:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829920.440025:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973221888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829920.440032:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973222016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829920.440048:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829920.440059:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff9c1a78a7a940 x1874668973222016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829920.440071:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973222016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829920.440079:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829920.440082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829920.440086:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973222016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829920.440091:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973222144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829920.440106:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829920.440115:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff9c1a78a79e00 x1874668973222144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829920.440126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973222144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829920.440134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829920.440138:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829920.440141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973222144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829920.440146:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973222272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787829920.440159:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829920.440167:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff9c1a78a783c0 x1874668973222272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829920.440178:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973222272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829920.440186:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829920.440189:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829920.440192:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973222272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829920.440207:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973222400:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829920.440221:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829920.440230:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff9c1a78a7a1c0 x1874668973222400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829920.440241:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973222400/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829920.440249:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829920.440252:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829920.440255:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973222400:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829920.440260:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973222528:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787829920.440273:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829920.440281:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829920/real 1787829920] req@ffff9c1a78a7b0c0 x1874668973222528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829920.440292:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973222528/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829920.440300:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829920.440303:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787829920.440306:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973222528:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829920.951455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829920.951459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829921.207503:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829921.975438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829921.975438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829922.231552:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829922.999486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829922.999493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829923.255571:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829924.023435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829924.023438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829924.279495:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829925.047444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829925.047444:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829925.303504:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829925.559449:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829925.559457:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829925.559462:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829925.559473:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6375 00000100:00080000:0.0:1787829925.559483:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829925.559491:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829925.559524:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973222656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829925.559549:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829925.559553:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829925.559555:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829925.559560:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6375 00000100:00080000:0.0:1787829925.559564:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829925.559569:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829925.559577:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973222656:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829925.559579:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829925.559597:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973222784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829925.559608:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829925.559612:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829925.559614:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829925.559618:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6375 00000100:00080000:0.0:1787829925.559622:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829925.559627:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829925.559632:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829925.559632:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829925.559647:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973222912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829925.559657:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829925.559661:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787829925.559663:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829925/real 1787829925] req@ffff9c1a78863c00 x1874668973222656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829925.559664:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829925.559673:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6375 00000100:00080000:0.0:1787829925.559678:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829925.559683:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829925.559688:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829925.559691:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973222656/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829925.559701:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973223040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829925.559704:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829925.559714:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829925.559717:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829925.559721:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973222656:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829925.559723:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829925.559726:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829925.559733:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6375 00000100:00080000:0.0:1787829925.559736:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829925.559740:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829925.559744:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829925.559749:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973222784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787829925.559755:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973223168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829925.559766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829925.559769:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829925.559772:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829925.559777:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6375 00000800:00000100:1.0:1787829925.559777:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829925.559784:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829925.559788:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829925.559794:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829925.559796:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829925/real 1787829925] req@ffff9c1a788612c0 x1874668973222784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829925.559807:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973223296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829925.559812:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973222784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829925.559823:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829925.559830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829925.559831:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829925.559838:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829925.559839:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973222784:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829925.559841:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829925.559849:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6375 00000100:00080000:0.0:1787829925.559854:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829925.559854:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973222912:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829925.559862:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829925.559868:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829925.559881:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973223424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829925.559887:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829925.559904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829925/real 1787829925] req@ffff9c1a78861e00 x1874668973222912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829925.559916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973222912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829925.559923:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829925.559927:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829925.559930:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973222912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829925.559938:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973223040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829925.559964:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829925.559978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829925/real 1787829925] req@ffff9c1a78861680 x1874668973223040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829925.559986:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973223040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829925.559992:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829925.559994:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829925.559997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973223040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829925.560012:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973223168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829925.560031:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829925.560041:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829925/real 1787829925] req@ffff9c1a78862580 x1874668973223168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829925.560051:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973223168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829925.560056:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829925.560058:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829925.560060:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973223168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829925.560064:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973223296:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829925.560076:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829925.560083:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829925/real 1787829925] req@ffff9c1a788630c0 x1874668973223296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829925.560089:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973223296/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829925.560093:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829925.560095:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829925.560096:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973223296:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829925.560099:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973223424:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829925.560108:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829925.560113:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829925/real 1787829925] req@ffff9c1a78860000 x1874668973223424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829925.560119:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973223424/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829925.560123:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829925.560124:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829925.560126:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973223424:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829926.071490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829926.071511:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829926.327577:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829927.095441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829927.095441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829927.351589:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829928.119451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829928.119451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829928.375531:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829929.143459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829929.143459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829929.399531:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829930.167463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829930.167463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829930.423464:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829930.679482:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829930.679493:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829930.679497:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829930.679508:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6381 00000100:00080000:0.0:1787829930.679522:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829930.679531:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829930.679581:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973223552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829930.679614:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829930.679618:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829930.679620:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829930.679624:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6381 00000100:00080000:0.0:1787829930.679629:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829930.679634:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829930.679646:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829930.679661:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973223680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829930.679673:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973223552:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829930.679678:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829930.679687:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829930.679690:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829930.679695:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6381 00000100:00080000:0.0:1787829930.679700:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829930.679705:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829930.679711:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829930.679723:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973223808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829930.679732:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829930.679735:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829930.679736:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829930.679739:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6381 00000100:00080000:0.0:1787829930.679743:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829930.679750:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829930.679756:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829930.679762:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829930.679774:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973223936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829930.679782:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829930.679785:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829930.679787:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829930.679792:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6381 00000100:00080000:0.0:1787829930.679797:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829930.679801:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829930/real 1787829930] req@ffff9c1a78860780 x1874668973223552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829930.679808:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829930.679821:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829930.679834:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973223552/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829930.679836:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973224064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829930.679875:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829930.679887:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829930.679890:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829930.679892:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829930.679894:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6381 00000100:00080000:0.0:1787829930.679903:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829930.679908:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787829930.679913:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829930.679917:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973223552:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829930.679921:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829930.679930:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973224192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829930.679943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973223680:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829930.679957:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829930.679961:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829930.679962:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829930.679967:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6381 00000100:00080000:0.0:1787829930.679976:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829930.679979:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829930.679983:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829930.679985:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787829930.679997:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973224320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829930.680009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829930/real 1787829930] req@ffff9c1a78861a40 x1874668973223680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829930.680020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973223680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829930.680026:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829930.680029:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829930.680032:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973223680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829930.680037:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973223808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829930.680052:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829930.680059:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829930/real 1787829930] req@ffff9c1a78862940 x1874668973223808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829930.680065:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973223808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829930.680070:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829930.680072:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829930.680074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973223808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829930.680077:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973223936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829930.680085:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829930.680090:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829930/real 1787829930] req@ffff9c1a78860b40 x1874668973223936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829930.680101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973223936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829930.680105:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829930.680107:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829930.680109:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973223936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829930.680112:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973224064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829930.680121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829930.680125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829930/real 1787829930] req@ffff9c1a788621c0 x1874668973224064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829930.680132:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973224064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829930.680136:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829930.680137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829930.680139:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973224064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829930.680150:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973224192:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829930.680160:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829930.680169:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829930/real 1787829930] req@ffff9c1a78863840 x1874668973224192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829930.680179:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973224192/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829930.680191:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829930.680194:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829930.680197:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973224192:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829930.680203:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973224320:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829930.680219:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829930.680227:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829930/real 1787829930] req@ffff9c1a78862d00 x1874668973224320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829930.680233:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973224320/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829930.680238:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829930.680240:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829930.680242:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973224320:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829931.191452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829931.191452:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829931.447527:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829932.215443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829932.215443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829932.471513:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829933.239445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829933.239445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829933.495496:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829934.263463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829934.263464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829934.519503:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829935.287467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829935.287467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829935.543792:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829935.799442:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829935.799453:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829935.799456:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829935.799465:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6386 00000100:00080000:2.0:1787829935.799476:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829935.799483:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787829935.799523:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973224448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829935.799550:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829935.799553:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829935.799563:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829935.799569:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6386 00000100:00080000:2.0:1787829935.799576:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829935.799581:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829935.799590:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829935.799599:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973224448:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787829935.799605:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973224576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829935.799617:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829935.799620:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829935.799622:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829935.799625:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6386 00000100:00080000:2.0:1787829935.799630:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829935.799634:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829935.799639:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829935.799651:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973224704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829935.799660:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829935.799667:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829935.799672:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829935.799674:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829935.799678:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6386 00000100:00080000:2.0:1787829935.799683:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829935.799687:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829935.799692:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829935.799700:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829935/real 1787829935] req@ffff9c1a49d8d680 x1874668973224448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829935.799722:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973224832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829935.799725:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973224448/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829935.799750:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829935.799750:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829935.799756:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829935.799759:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973224448:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787829935.799765:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829935.799768:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829935.799773:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6386 00000100:00080000:2.0:1787829935.799777:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829935.799780:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829935.799786:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829935.799787:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973224576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787829935.799801:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973224960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829935.799812:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829935.799817:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787829935.799817:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829935.799822:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829935.799828:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6386 00000100:00080000:2.0:1787829935.799833:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829935.799837:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829935/real 1787829935] req@ffff9c1a49d8e580 x1874668973224576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829935.799840:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829935.799849:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829935.799858:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973224576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829935.799864:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973225088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829935.799873:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829935.799881:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829935.799888:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973224576:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829935.799891:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829935.799896:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829935.799898:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829935.799899:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973224704:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787829935.799904:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6386 00000100:00080000:2.0:1787829935.799910:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829935.799914:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787829935.799918:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829935.799928:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787829935.799933:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973225216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829935.799943:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829935/real 1787829935] req@ffff9c1a49d8c3c0 x1874668973224704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829935.799957:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973224704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829935.799970:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829935.799974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829935.799977:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973224704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829935.799983:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973224832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829935.800002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829935.800011:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829935/real 1787829935] req@ffff9c1a49d8f0c0 x1874668973224832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829935.800018:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973224832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829935.800023:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829935.800025:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829935.800027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973224832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829935.800038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973224960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829935.800055:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829935.800064:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829935/real 1787829935] req@ffff9c1a49d8ed00 x1874668973224960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829935.800075:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973224960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829935.800083:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829935.800086:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829935.800090:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973224960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829935.800095:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973225088:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829935.800113:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829935.800123:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829935/real 1787829935] req@ffff9c1a49d8fc00 x1874668973225088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829935.800135:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973225088/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829935.800142:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829935.800143:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829935.800146:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973225088:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829935.800151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973225216:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829935.800168:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829935.800178:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829935/real 1787829935] req@ffff9c1a49d8e1c0 x1874668973225216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829935.800190:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973225216/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829935.800203:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829935.800207:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829935.800209:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973225216:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829936.311408:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829936.311412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829936.567480:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829937.335443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829937.335456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829937.591512:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829938.359454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829938.359454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829938.615508:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829939.383443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829939.383444:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829939.639516:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829940.407453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829940.407470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829940.663476:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829940.919475:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829940.919484:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829940.919487:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829940.919496:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6391 00000100:00080000:3.0:1787829940.919507:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829940.919519:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829940.919554:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973225344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829940.919579:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829940.919583:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829940.919588:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829940.919592:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6391 00000100:00080000:3.0:1787829940.919596:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829940.919601:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829940.919610:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829940.919621:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973225344:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829940.919626:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973225472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829940.919643:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829940.919647:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829940.919649:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829940.919653:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6391 00000100:00080000:3.0:1787829940.919657:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829940.919661:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829940.919666:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829940.919670:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787829940.919682:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973225600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829940.919693:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829940.919697:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829940.919699:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829940.919703:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829940/real 1787829940] req@ffff9c1a7895c3c0 x1874668973225344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829940.919705:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6391 00000100:00080000:3.0:1787829940.919711:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829940.919718:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829940.919722:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973225344/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829940.919724:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829940.919734:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829940.919739:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787829940.919740:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973225728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829940.919746:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973225344:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829940.919754:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829940.919758:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829940.919759:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829940.919763:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6391 00000100:00080000:3.0:1787829940.919767:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829940.919771:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829940.919775:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973225472:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829940.919776:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829940.919791:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973225856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829940.919801:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829940.919806:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787829940.919807:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829940.919809:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829940.919815:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6391 00000100:00080000:3.0:1787829940.919818:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829940.919822:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829940.919825:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829940/real 1787829940] req@ffff9c1a7895c780 x1874668973225472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829940.919828:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829940.919841:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973225984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829940.919844:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973225472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829940.919866:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829940.919868:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829940.919870:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787829940.919873:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829940.919876:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973225472:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829940.919877:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829940.919883:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6391 00000100:00100000:1.0:1787829940.919888:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973225600:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787829940.919888:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829940.919895:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829940.919901:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829940.919911:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973226112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829940.919925:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829940.919938:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829940/real 1787829940] req@ffff9c1a7895e940 x1874668973225600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829940.919946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973225600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829940.919951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829940.919953:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829940.919955:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973225600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829940.919958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973225728:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829940.919971:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829940.919976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829940/real 1787829940] req@ffff9c1a7895cf00 x1874668973225728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829940.919984:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973225728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829940.919998:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829940.920001:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829940.920004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973225728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829940.920018:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973225856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829940.920035:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829940.920043:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829940/real 1787829940] req@ffff9c1a7895e580 x1874668973225856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829940.920050:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973225856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829940.920056:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829940.920059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829940.920062:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973225856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829940.920067:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973225984:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829940.920084:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829940.920095:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829940/real 1787829940] req@ffff9c1a7895c000 x1874668973225984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829940.920112:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973225984/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829940.920117:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829940.920119:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829940.920121:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973225984:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829940.920125:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973226112:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829940.920138:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829940.920145:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829940/real 1787829940] req@ffff9c1a7895f480 x1874668973226112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829940.920151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973226112/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829940.920159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829940.920161:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829940.920163:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973226112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829941.431420:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829941.431424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829941.687497:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829942.455451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829942.455451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829942.711475:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829943.479447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829943.479447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829943.735566:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829944.503446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829944.503446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829944.759514:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829945.527465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829945.527465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829945.783511:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829946.039472:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829946.039479:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829946.039482:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829946.039489:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6396 00000100:00080000:3.0:1787829946.039499:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829946.039506:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829946.039540:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973226240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829946.039561:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829946.039563:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829946.039564:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829946.039567:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6396 00000100:00080000:3.0:1787829946.039570:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829946.039572:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829946.039579:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829946.039588:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973226368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829946.039596:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829946.039598:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829946.039599:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829946.039601:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6396 00000100:00080000:3.0:1787829946.039604:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829946.039605:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973226240:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829946.039608:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829946.039615:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829946.039621:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973226496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829946.039627:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829946.039629:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829946.039630:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829946.039632:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6396 00000100:00080000:3.0:1787829946.039634:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829946.039636:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829946.039639:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829946.039644:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973226624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829946.039649:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829946.039651:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829946.039652:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829946.039660:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6396 00000100:00080000:3.0:1787829946.039663:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829946.039665:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829946.039667:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829946.039673:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973226752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829946.039678:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829946.039679:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829946.039680:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829946.039682:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6396 00000100:00080000:3.0:1787829946.039684:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829946.039685:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829946.039688:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829946.039692:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973226880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829946.039696:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829946.039698:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829946.039699:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829946.039700:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6396 00000100:00080000:3.0:1787829946.039702:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829946.039704:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829946.039706:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829946.039711:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973227008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829946.040706:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829946.040743:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff9c1a7895d2c0 x1874668973226240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829946.040760:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973226240/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829946.040770:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829946.040774:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829946.040778:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973226240:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829946.040786:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973226368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829946.040809:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829946.040822:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff9c1a7895cb40 x1874668973226368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829946.040841:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973226368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829946.040849:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829946.040852:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829946.040855:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973226368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829946.040875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973226496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829946.040893:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829946.040902:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff9c1a7895e1c0 x1874668973226496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829946.040913:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973226496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829946.040920:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829946.040924:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829946.040927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973226496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829946.040931:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973226624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829946.040944:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829946.040966:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff9c1a7895da40 x1874668973226624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829946.040977:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973226624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829946.040991:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829946.040993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829946.040995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973226624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829946.040999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973226752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829946.041013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829946.041020:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff9c1a7895de00 x1874668973226752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829946.041027:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973226752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829946.041031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829946.041034:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829946.041037:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973226752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829946.041042:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973226880:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829946.041057:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829946.041064:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff9c1a7895f840 x1874668973226880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829946.041070:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973226880/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829946.041075:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829946.041078:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829946.041081:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973226880:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829946.041087:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973227008:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829946.041103:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829946.041113:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829946/real 1787829946] req@ffff9c1a7895f0c0 x1874668973227008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829946.041124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973227008/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829946.041131:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829946.041134:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829946.041137:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973227008:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829946.551457:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829946.551457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829946.807489:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829947.575422:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829947.575422:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829947.831520:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829948.599439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829948.599443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829948.855524:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829949.623438:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829949.623438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829949.879453:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829950.647418:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829950.647448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829950.903601:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829951.159517:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.159561:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829951.159566:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.159576:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6401 00000100:00080000:0.0:1787829951.159588:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.159596:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829951.159633:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973227136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829951.159662:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.159666:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829951.159668:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.159673:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6401 00000100:00080000:0.0:1787829951.159687:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.159692:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829951.159702:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829951.159714:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973227136:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829951.159717:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973227264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829951.159728:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.159731:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829951.159732:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.159735:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6401 00000100:00080000:0.0:1787829951.159738:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.159741:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829951.159745:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829951.159753:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973227392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829951.159761:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.159763:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829951.159765:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.159767:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6401 00000100:00080000:0.0:1787829951.159769:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.159773:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787829951.159775:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:0.0:1787829951.159777:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829951.159796:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973227520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829951.159804:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.159806:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829951.159808:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.159811:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6401 00000100:00080000:0.0:1787829951.159814:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.159817:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829951.159820:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829951.159824:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff9c1a78860f00 x1874668973227136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829951.159829:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973227648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829951.159848:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.159850:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829951.159852:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829951.159856:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973227136/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829951.159858:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6401 00000100:00080000:0.0:1787829951.159863:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.159869:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829951.159874:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829951.159874:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.159882:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787829951.159886:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973227776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829951.159889:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973227136:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787829951.159902:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829951.159904:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829951.159906:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829951.159909:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6401 00000100:00080000:0.0:1787829951.159911:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829951.159914:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829951.159917:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973227264:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787829951.159919:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829951.159931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973227904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829951.159942:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829951.159955:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff9c1a78862d00 x1874668973227264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.159965:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973227264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.159977:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.159980:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829951.159982:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973227264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829951.159990:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973227392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829951.160003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829951.160009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff9c1a78863840 x1874668973227392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.160022:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973227392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.160044:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.160048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829951.160052:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973227392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829951.160059:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973227520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829951.160081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829951.160090:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff9c1a788621c0 x1874668973227520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.160098:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973227520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.160114:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.160116:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829951.160118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973227520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829951.160123:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973227648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829951.160134:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829951.160140:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff9c1a78860b40 x1874668973227648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.160147:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973227648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.160153:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.160154:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829951.160156:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973227648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829951.160159:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973227776:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829951.160170:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829951.160175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff9c1a78862940 x1874668973227776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.160187:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973227776/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.160207:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.160209:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829951.160212:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973227776:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829951.160225:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973227904:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829951.160241:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829951.160248:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829951/real 1787829951] req@ffff9c1a78861a40 x1874668973227904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829951.160256:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973227904/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829951.160261:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829951.160263:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829951.160265:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973227904:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829951.671469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829951.671469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829951.927567:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829952.695461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829952.695461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829952.951528:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829953.719464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829953.719464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829953.975513:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829954.743449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829954.743449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829955.000555:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829955.767442:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829955.767442:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829956.023501:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829956.279419:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829956.279432:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829956.279435:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829956.279443:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6406 00000100:00080000:0.0:1787829956.279451:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829956.279457:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829956.279485:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973228032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829956.279505:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829956.279508:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829956.279509:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829956.279543:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6406 00000100:00080000:0.0:1787829956.279546:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829956.279549:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829956.279556:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973228032:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787829956.279557:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829956.279570:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973228160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829956.279577:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829956.279579:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829956.279580:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829956.279588:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6406 00000100:00080000:0.0:1787829956.279597:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829956.279600:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829956.279603:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829956.279612:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973228288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829956.279612:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829956.279621:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829956.279624:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829956.279625:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829956.279627:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6406 00000100:00080000:0.0:1787829956.279629:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829956.279631:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829956.279634:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829956.279640:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973228416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829956.279645:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829956.279647:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829956.279648:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829956.279650:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6406 00000100:00080000:0.0:1787829956.279652:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829956.279656:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829956.279657:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff9c1a78860780 x1874668973228032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829956.279659:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829956.279670:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973228544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829956.279684:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829956.279687:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973228032/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829956.279688:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829956.279691:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829956.279697:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6406 00000100:00080000:0.0:1787829956.279700:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829956.279701:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829956.279704:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787829956.279708:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:0.0:1787829956.279709:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829956.279712:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973228032:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787829956.279719:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973228672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829956.279726:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829956.279727:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829956.279729:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829956.279731:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6406 00000100:00080000:0.0:1787829956.279733:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829956.279735:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973228160:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829956.279737:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829956.279743:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829956.279750:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973228800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829956.279763:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829956.279772:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff9c1a78860000 x1874668973228160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829956.279782:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973228160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829956.279787:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829956.279789:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829956.279791:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973228160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829956.279795:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973228288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829956.279805:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829956.279810:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff9c1a788630c0 x1874668973228288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829956.279817:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973228288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829956.279825:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829956.279827:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829956.279830:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973228288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829956.279836:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973228416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829956.279851:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829956.279861:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff9c1a78862580 x1874668973228416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829956.279873:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973228416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829956.279881:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829956.279883:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829956.279884:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973228416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829956.279888:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973228544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829956.279901:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829956.279908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff9c1a78861680 x1874668973228544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829956.279914:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973228544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829956.279919:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829956.279921:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829956.279923:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973228544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829956.279926:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973228672:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829956.279936:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829956.279940:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff9c1a78861e00 x1874668973228672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829956.279953:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973228672/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829956.279958:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829956.279960:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829956.279961:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973228672:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829956.279973:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973228800:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829956.279982:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829956.279988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829956/real 1787829956] req@ffff9c1a788612c0 x1874668973228800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829956.279994:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973228800/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829956.280004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829956.280006:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829956.280008:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973228800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829956.791458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829956.791490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829957.047516:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829957.815424:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829957.815425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829958.071485:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829958.839485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829958.839492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829959.095507:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829959.863496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829959.863496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829960.119502:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829960.887454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829960.887454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829961.143550:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829961.400220:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829961.400228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829961.400236:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829961.400246:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6411 00000100:00080000:0.0:1787829961.400255:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829961.400262:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787829961.400289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973228928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829961.400368:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973228928:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787829961.400421:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829961.400467:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829961.400471:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829961.400473:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829961.400477:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6411 00000100:00080000:0.0:1787829961.400480:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829961.400484:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829961.400492:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829961.400504:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973229056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829961.400780:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff9c1a78863c00 x1874668973228928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829961.400804:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973228928/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829961.400814:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829961.400818:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829961.400821:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973228928:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829961.400845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973229056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829961.400865:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829961.400875:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff9c1a788612c0 x1874668973229056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829961.400886:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973229056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829961.400894:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829961.400897:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829961.400900:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973229056:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829961.401158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829961.401163:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829961.401165:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829961.401169:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6411 00000100:00080000:0.0:1787829961.401173:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829961.401177:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829961.401182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829961.401194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973229184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829961.401239:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973229184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829961.401262:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829961.401284:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829961.401288:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829961.401289:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829961.401293:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6411 00000100:00080000:0.0:1787829961.401297:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829961.401301:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829961.401305:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829961.401315:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973229312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829961.401364:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff9c1a78861e00 x1874668973229184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829961.401377:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973229184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829961.401385:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829961.401387:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829961.401391:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973229184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829961.401401:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973229312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829961.401419:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829961.401429:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff9c1a78861680 x1874668973229312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829961.401439:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973229312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829961.401447:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829961.401450:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829961.401453:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973229312:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787829961.401538:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829961.401542:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829961.401544:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829961.401547:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6411 00000100:00080000:0.0:1787829961.401551:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829961.401554:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829961.401559:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829961.401569:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973229440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829961.401605:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973229440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829961.401632:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787829961.401648:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829961.401651:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829961.401653:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829961.401657:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6411 00000100:00080000:0.0:1787829961.401660:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829961.401664:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829961.401668:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829961.401678:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973229568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829961.401755:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff9c1a78862580 x1874668973229440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829961.401768:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973229440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829961.401776:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829961.401779:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829961.401782:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973229440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829961.401792:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973229568:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829961.401810:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829961.401820:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff9c1a788630c0 x1874668973229568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829961.401829:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973229568/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829961.401837:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829961.401839:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829961.401842:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973229568:192.168.203.148@tcp:8:kworker.0 00000100:00080000:0.0:1787829961.401924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829961.401928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829961.401930:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829961.401934:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6411 00000100:00080000:0.0:1787829961.401938:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829961.401942:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787829961.401947:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829961.401959:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973229696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829961.401998:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973229696:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829961.402021:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829961.402169:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829961/real 1787829961] req@ffff9c1a78860000 x1874668973229696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829961.402180:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973229696/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829961.402187:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829961.402189:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829961.402193:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973229696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829961.911449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829961.911449:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829962.167538:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829962.935446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829962.935446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829963.191562:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829963.959459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829963.959464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829964.215530:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829964.983484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829964.983484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829965.239798:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829966.007501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829966.007501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829966.263559:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829966.519919:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829966.519929:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829966.519933:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829966.519942:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6416 00000100:00080000:3.0:1787829966.519952:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829966.519959:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829966.519993:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973229824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829966.520021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829966.520024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829966.520026:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829966.520030:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6416 00000100:00080000:3.0:1787829966.520034:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829966.520037:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829966.520045:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829966.520058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973229952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829966.520067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829966.520070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829966.520071:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829966.520075:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6416 00000100:00080000:3.0:1787829966.520077:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829966.520080:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829966.520095:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829966.520105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973230080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829966.520114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829966.520117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829966.520119:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829966.520122:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6416 00000100:00080000:3.0:1787829966.520125:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829966.520128:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829966.520132:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829966.520140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973230208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829966.520148:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829966.520152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829966.520153:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829966.520157:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6416 00000100:00080000:3.0:1787829966.520160:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829966.520164:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973229824:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829966.520165:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829966.520171:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829966.520179:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973230336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829966.520188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829966.520191:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829966.520194:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829966.520197:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6416 00000100:00080000:3.0:1787829966.520200:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829966.520203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829966.520207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829966.520217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973230464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829966.520225:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829966.520228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829966.520229:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829966.520233:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6416 00000100:00080000:3.0:1787829966.520236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829966.520238:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829966.520243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829966.520253:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973230592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829966.520261:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973229952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829966.520295:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973230080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829966.520317:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973230208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829966.520386:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973230336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829966.520411:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973230464:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829966.520435:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973230592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829967.031458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829967.031458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829967.287463:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829968.055485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829968.055490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829968.311680:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829969.079438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829969.079454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829969.335732:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829969.591643:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787829969.591689:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829969.591704:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829969.591741:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829969.591749:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829969.591756:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829969.591762:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829969.591768:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787829969.591775:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787829969.591857:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829966/real 1787829969] req@ffff9c1a7895da40 x1874668973230336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829969.591880:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973230336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829969.591892:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829969.591897:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829969.591900:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973230336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829969.591907:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829966/real 1787829969] req@ffff9c1a7895e1c0 x1874668973230464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829969.591918:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973230464/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829969.591925:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829969.591928:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829969.591931:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973230464:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829969.591938:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829966/real 1787829969] req@ffff9c1a7895cb40 x1874668973230592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829969.591949:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973230592/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829969.591956:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829969.591958:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829969.591961:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973230592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829969.591968:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829966/real 1787829969] req@ffff9c1a7895fc00 x1874668973229824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829969.591981:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973229824/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829969.591989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829969.591993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829969.591997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973229824:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829969.592002:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829966/real 1787829969] req@ffff9c1a7895f0c0 x1874668973229952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829969.592012:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973229952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829969.592018:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829969.592021:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829969.592024:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973229952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829969.592029:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829966/real 1787829969] req@ffff9c1a7895f840 x1874668973230080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829969.592039:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973230080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829969.592045:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829969.592048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829969.592050:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973230080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829969.592056:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829966/real 1787829969] req@ffff9c1a7895de00 x1874668973230208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829969.592080:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973230208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829969.592088:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829969.592092:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829969.592095:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973230208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829970.103495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829970.103607:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829970.359601:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829971.127451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829971.127560:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829971.383616:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829971.639443:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829971.639454:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829971.639458:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829971.639467:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6421 00000100:00080000:3.0:1787829971.639477:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829971.639484:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829971.639514:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973230720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829971.639547:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829971.639550:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829971.639552:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829971.639555:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6421 00000100:00080000:3.0:1787829971.639559:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829971.639562:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829971.639570:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829971.639589:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973230848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829971.639598:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829971.639600:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829971.639602:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829971.639605:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6421 00000100:00080000:3.0:1787829971.639608:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829971.639611:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829971.639615:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829971.639623:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973230976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829971.639638:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829971.639641:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829971.639642:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829971.639646:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6421 00000100:00080000:3.0:1787829971.639648:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829971.639652:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829971.639655:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829971.639664:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973231104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829971.639672:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829971.639675:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829971.639677:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829971.639680:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6421 00000100:00080000:3.0:1787829971.639683:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829971.639686:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829971.639690:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829971.639699:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973231232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829971.639707:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829971.639710:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829971.639711:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829971.639715:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6421 00000100:00080000:3.0:1787829971.639718:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829971.639721:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829971.639726:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829971.639733:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973231360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829971.639742:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829971.639745:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829971.639746:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829971.639749:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6421 00000100:00080000:3.0:1787829971.639752:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829971.639754:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829971.639758:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829971.639765:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973231488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829971.639851:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973230720:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787829971.639907:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829971.639940:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829971/real 1787829971] req@ffff9c1a7895d2c0 x1874668973230720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829971.639955:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973230720/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829971.639965:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829971.639968:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829971.639972:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973230720:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829971.639980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973230848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829971.640002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829971.640012:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829971/real 1787829971] req@ffff9c1a7895f480 x1874668973230848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829971.640022:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973230848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829971.640028:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829971.640031:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829971.640034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973230848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829971.640040:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973230976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829971.640068:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829971.640079:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829971/real 1787829971] req@ffff9c1a7895c000 x1874668973230976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829971.640090:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973230976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829971.640097:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829971.640100:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829971.640102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973230976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829971.640107:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973231104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829971.640123:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829971.640133:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829971/real 1787829971] req@ffff9c1a7895e580 x1874668973231104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829971.640142:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973231104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829971.640149:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829971.640152:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829971.640155:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973231104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829971.640159:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973231232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829971.640174:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829971.640184:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829971/real 1787829971] req@ffff9c1a7895cf00 x1874668973231232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829971.640195:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973231232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829971.640202:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829971.640205:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829971.640207:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973231232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829971.640212:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973231360:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829971.640229:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829971.640238:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829971/real 1787829971] req@ffff9c1a7895e940 x1874668973231360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829971.640255:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973231360/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829971.640264:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829971.640267:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829971.640269:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973231360:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829971.640274:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973231488:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829971.640291:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829971.640301:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829971/real 1787829971] req@ffff9c1a7895c780 x1874668973231488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829971.640316:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973231488/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829971.640323:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829971.640326:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829971.640328:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973231488:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787829972.151568:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829972.152086:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829972.407726:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829973.175483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829973.175568:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829973.431549:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829974.199482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829974.199482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829974.455553:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829975.223430:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829975.223430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829975.479654:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829976.247460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829976.247460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829976.503565:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829976.759466:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829976.759476:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829976.759480:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829976.759491:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6427 00000100:00080000:3.0:1787829976.759501:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829976.759508:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829976.759543:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973231616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829976.759583:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829976.759587:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829976.759589:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829976.759593:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6427 00000100:00080000:3.0:1787829976.759596:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829976.759600:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829976.759609:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829976.759622:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973231744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829976.759631:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829976.759634:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829976.759636:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829976.759639:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6427 00000100:00080000:3.0:1787829976.759642:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829976.759646:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829976.759650:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829976.759659:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973231872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829976.759668:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829976.759671:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829976.759673:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829976.759676:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6427 00000100:00080000:3.0:1787829976.759679:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829976.759683:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829976.759687:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829976.759696:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973232000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829976.759703:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829976.759706:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829976.759708:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829976.759711:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6427 00000100:00080000:3.0:1787829976.759714:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829976.759717:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829976.759721:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829976.759730:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973232128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829976.759738:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829976.759741:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829976.759742:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829976.759746:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6427 00000100:00080000:3.0:1787829976.759749:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829976.759752:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829976.759757:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829976.759765:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973232256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829976.759779:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829976.759782:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829976.759784:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829976.759787:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6427 00000100:00080000:3.0:1787829976.759790:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829976.759793:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829976.759796:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829976.759804:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973232384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829976.760007:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973231616:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787829976.760078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829976.760169:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829976/real 1787829976] req@ffff9c1a7895c3c0 x1874668973231616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829976.760187:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973231616/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829976.760198:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829976.760201:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829976.760206:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973231616:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829976.760213:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973231744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829976.760237:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829976.760249:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829976/real 1787829976] req@ffff9c1a7895c780 x1874668973231744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829976.760260:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973231744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829976.760268:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829976.760271:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829976.760274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973231744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829976.760279:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973231872:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829976.760295:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829976.760304:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829976/real 1787829976] req@ffff9c1a7895e940 x1874668973231872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829976.760314:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973231872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829976.760321:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829976.760324:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829976.760327:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973231872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829976.760333:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973232000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829976.760375:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829976.760385:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829976/real 1787829976] req@ffff9c1a7895cf00 x1874668973232000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829976.760400:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973232000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829976.760407:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829976.760409:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829976.760412:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973232000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829976.760417:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973232128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829976.760432:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829976.760440:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829976/real 1787829976] req@ffff9c1a7895e580 x1874668973232128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829976.760449:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973232128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829976.760455:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829976.760458:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829976.760460:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973232128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829976.760464:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973232256:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829976.760478:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829976.760485:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829976/real 1787829976] req@ffff9c1a7895c000 x1874668973232256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829976.760493:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973232256/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829976.760504:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829976.760507:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829976.760509:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973232256:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829976.760514:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973232384:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829976.760528:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829976.760537:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829976/real 1787829976] req@ffff9c1a7895f480 x1874668973232384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829976.760547:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973232384/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829976.760554:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829976.760557:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829976.760560:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973232384:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829977.271464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829977.271603:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829977.527687:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829978.295602:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829978.295638:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829978.551463:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829979.319518:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829979.321112:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829979.575568:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829980.343419:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829980.343500:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829980.600394:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829981.368651:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829981.368778:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829981.623752:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829981.879486:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829981.879497:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829981.879502:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829981.879515:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6432 00000100:00080000:3.0:1787829981.879528:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829981.879536:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829981.879580:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973232512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829981.879615:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829981.879620:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829981.879621:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829981.879628:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6432 00000100:00080000:3.0:1787829981.879631:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829981.879635:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829981.879646:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829981.879657:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973232640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829981.879665:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829981.879668:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829981.879670:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829981.879674:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6432 00000100:00080000:3.0:1787829981.879676:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829981.879679:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829981.879684:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829981.879694:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973232768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829981.879705:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829981.879709:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829981.879710:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829981.879714:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6432 00000100:00080000:3.0:1787829981.879717:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829981.879721:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829981.879726:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829981.879736:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973232896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829981.879748:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973232512:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787829981.879751:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829981.879761:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829981.879768:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829981.879773:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6432 00000100:00080000:3.0:1787829981.879776:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829981.879780:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829981.879785:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829981.879797:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973233024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829981.879804:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829981.879807:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829981.879809:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829981.879812:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6432 00000100:00080000:3.0:1787829981.879818:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829981.879822:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829981.879827:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829981.879839:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973233152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829981.879843:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829981.879856:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829981.879859:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829981.879862:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829981.879867:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6432 00000100:00080000:3.0:1787829981.879871:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829981.879876:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829981.879881:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829981.879894:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973233280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829981.879898:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829981/real 1787829981] req@ffff9c1a7895d2c0 x1874668973232512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829981.879937:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973232512/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829981.879949:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829981.879953:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829981.879957:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973232512:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829981.879965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973232640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829981.880003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829981.880015:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829981/real 1787829981] req@ffff9c1a7895de00 x1874668973232640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829981.880026:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973232640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829981.880033:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829981.880035:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829981.880038:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973232640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829981.880055:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973232768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829981.880073:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829981.880090:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829981/real 1787829981] req@ffff9c1a7895f840 x1874668973232768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829981.880101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973232768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829981.880114:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829981.880117:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829981.880120:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973232768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829981.880145:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973232896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829981.880165:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829981.880176:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829981/real 1787829981] req@ffff9c1a7895f0c0 x1874668973232896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829981.880187:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973232896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829981.880194:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829981.880197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829981.880200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973232896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829981.880206:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973233024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829981.880224:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829981.880245:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829981/real 1787829981] req@ffff9c1a7895fc00 x1874668973233024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829981.880257:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973233024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829981.880264:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829981.880267:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829981.880271:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973233024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829981.880277:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973233152:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829981.880296:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829981.880307:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829981/real 1787829981] req@ffff9c1a7895cb40 x1874668973233152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829981.880317:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973233152/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829981.880325:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829981.880328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829981.880330:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973233152:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829981.880337:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973233280:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829981.880402:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829981.880411:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829981/real 1787829981] req@ffff9c1a7895e1c0 x1874668973233280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829981.880419:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973233280/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829981.880423:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829981.880425:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829981.880427:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973233280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829982.391709:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829982.392117:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829982.647891:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829983.415468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829983.415468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829983.671520:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829984.439468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829984.439474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829984.695473:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829985.463495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829985.463495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829985.719674:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829986.487437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829986.487483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829986.743566:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829986.999460:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829986.999474:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829986.999480:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829986.999493:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6437 00000100:00080000:3.0:1787829986.999506:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829986.999514:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829986.999559:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973233408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829986.999599:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829986.999603:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829986.999605:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829986.999610:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6437 00000100:00080000:3.0:1787829986.999614:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829986.999618:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829986.999629:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829986.999644:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973233536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829986.999653:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829986.999656:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829986.999657:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829986.999661:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6437 00000100:00080000:3.0:1787829986.999664:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829986.999667:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829986.999672:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829986.999791:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973233408:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829986.999794:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973233664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829986.999824:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829986.999829:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829986.999832:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829986.999840:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6437 00000100:00080000:3.0:1787829986.999851:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829986.999859:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787829986.999872:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829986.999872:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829986.999907:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973233792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829986.999920:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829986.999924:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829986.999925:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829986.999929:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829986/real 1787829986] req@ffff9c1a7895da40 x1874668973233408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829986.999934:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6437 00000100:00080000:3.0:1787829986.999959:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829986.999970:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829986.999977:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829986.999985:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973233408/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829987.000002:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973233920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829987.000012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829987.000016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829987.000017:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829987.000021:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6437 00000100:00080000:3.0:1787829987.000024:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829987.000028:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829987.000032:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829987.000040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973234048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829987.000048:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829987.000051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829987.000053:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829987.000056:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6437 00000100:00080000:3.0:1787829987.000060:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829987.000064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829987.000065:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829987.000075:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829987.000077:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829987.000085:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973233408:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829987.000095:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973233536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787829987.000103:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973234176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829987.000127:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829987.000143:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829986/real 1787829987] req@ffff9c1a7895e1c0 x1874668973233536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829987.000157:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973233536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829987.000165:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829987.000169:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829987.000172:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973233536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829987.000200:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973233664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829987.000217:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829987.000227:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829986/real 1787829987] req@ffff9c1a7895cb40 x1874668973233664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829987.000245:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973233664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829987.000252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829987.000255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829987.000259:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973233664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829987.000264:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973233792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829987.000283:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829987.000293:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829986/real 1787829987] req@ffff9c1a7895fc00 x1874668973233792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829987.000561:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973233792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829987.000570:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829987.000573:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829987.000577:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973233792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829987.000583:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973233920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829987.000604:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829987.000614:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829987/real 1787829987] req@ffff9c1a7895f0c0 x1874668973233920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829987.000625:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973233920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829987.000639:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829987.000642:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829987.000645:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973233920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829987.000651:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973234048:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829987.000669:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829987.000679:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829987/real 1787829987] req@ffff9c1a7895f840 x1874668973234048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829987.000703:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973234048/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829987.000712:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829987.000715:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829987.000718:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973234048:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829987.000724:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973234176:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829987.000743:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829987.000753:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829987/real 1787829987] req@ffff9c1a7895de00 x1874668973234176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829987.000764:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973234176/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829987.000770:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829987.000774:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829987.000777:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973234176:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829987.511505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:2.0:1787829987.511737:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829987.767562:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829988.535416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829988.535655:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829988.791571:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829989.559456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829989.559456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829989.815521:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829990.583464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829990.584547:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829990.839730:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829991.607470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829991.608003:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829991.863523:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829992.119534:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.119544:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829992.119549:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.119558:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6442 00000100:00080000:3.0:1787829992.119569:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.119576:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829992.119612:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973234304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829992.119644:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.119648:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829992.119650:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.119654:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6442 00000100:00080000:3.0:1787829992.119658:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.119662:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829992.119671:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829992.119683:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973234432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829992.119692:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.119695:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829992.119697:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.119700:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6442 00000100:00080000:3.0:1787829992.119704:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.119707:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829992.119711:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829992.119720:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973234560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829992.119730:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.119734:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829992.119735:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.119739:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6442 00000100:00080000:3.0:1787829992.119742:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.119746:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829992.119750:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829992.119759:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973234688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829992.119768:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.119771:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829992.119773:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.119776:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6442 00000100:00080000:3.0:1787829992.119779:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.119782:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829992.119786:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829992.119796:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973234816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829992.119805:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.119808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829992.119810:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.119813:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6442 00000100:00080000:3.0:1787829992.119816:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.119820:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829992.119824:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829992.119833:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973234944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829992.119842:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829992.119845:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829992.119847:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829992.119850:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6442 00000100:00080000:3.0:1787829992.119853:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829992.119856:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829992.119860:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829992.119870:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973235072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829992.120004:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973234304:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787829992.120070:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829992.120102:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff9c1a7895d2c0 x1874668973234304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829992.120118:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973234304/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829992.120128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829992.120132:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829992.120135:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973234304:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787829992.120142:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973234432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829992.120163:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829992.120175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff9c1a7895f480 x1874668973234432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829992.120186:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973234432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829992.120195:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829992.120198:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829992.120202:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973234432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829992.120208:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973234560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829992.120240:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829992.120250:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff9c1a7895c000 x1874668973234560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829992.120263:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973234560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829992.120271:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829992.120274:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829992.120277:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973234560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829992.120283:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973234688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829992.120300:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829992.120310:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff9c1a7895e580 x1874668973234688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829992.120322:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973234688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829992.120329:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829992.120333:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829992.120336:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973234688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829992.120341:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973234816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829992.120387:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829992.120410:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff9c1a7895cf00 x1874668973234816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829992.120423:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973234816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829992.120431:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829992.120434:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829992.120438:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973234816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829992.120443:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973234944:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829992.120462:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829992.120472:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff9c1a7895e940 x1874668973234944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829992.120484:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973234944/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829992.120492:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829992.120495:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829992.120499:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973234944:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829992.120504:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973235072:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829992.120699:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829992.120711:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829992/real 1787829992] req@ffff9c1a7895c780 x1874668973235072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829992.120723:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973235072/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829992.120731:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829992.120735:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829992.120738:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973235072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787829992.631496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829992.631566:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829992.887713:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829993.655443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829993.655443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829993.911656:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829994.679465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829994.679499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829994.935547:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829995.703427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829995.703427:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829995.959556:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829996.727435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829996.727435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829996.983447:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829997.239433:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829997.239440:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829997.239444:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829997.239451:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6447 00000100:00080000:3.0:1787829997.239460:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829997.239465:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787829997.239496:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973235200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829997.239519:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829997.239523:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829997.239525:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829997.239530:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6447 00000100:00080000:3.0:1787829997.239534:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829997.239540:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829997.239541:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973235200:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787829997.239550:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829997.239568:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973235328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829997.239581:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829997.239583:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829997.239585:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829997.239587:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6447 00000100:00080000:3.0:1787829997.239590:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829997.239593:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787829997.239596:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829997.239601:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829997.239610:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973235456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829997.239617:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829997.239619:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829997.239620:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829997.239623:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6447 00000100:00080000:3.0:1787829997.239625:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829997.239627:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787829997.239630:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829997.239633:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff9c1a7895c3c0 x1874668973235200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829997.239640:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973235584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829997.239648:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829997.239650:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829997.239652:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829997.239653:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973235200/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829997.239656:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6447 00000100:00080000:3.0:1787829997.239661:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829997.239667:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787829997.239669:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829997.239670:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829997.239674:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829997.239679:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973235200:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787829997.239680:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973235712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829997.239688:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829997.239690:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829997.239691:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829997.239694:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6447 00000100:00080000:3.0:1787829997.239697:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829997.239700:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787829997.239704:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973235328:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787829997.239704:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829997.239713:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973235840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829997.239719:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829997.239722:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829997.239723:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829997.239725:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6447 00000100:00080000:3.0:1787829997.239729:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829997.239731:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:1.0:1787829997.239736:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787829997.239737:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829997.239746:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973235968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829997.239755:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff9c1a7895c780 x1874668973235328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829997.239778:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973235328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829997.239784:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829997.239786:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829997.239789:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973235328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829997.239794:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973235456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829997.239818:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829997.239831:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff9c1a7895e940 x1874668973235456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829997.239843:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973235456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829997.239851:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829997.239854:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829997.239858:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973235456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829997.239865:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973235584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829997.239902:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829997.239917:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff9c1a7895cf00 x1874668973235584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829997.239931:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973235584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829997.239943:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829997.239946:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829997.239948:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973235584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829997.239953:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973235712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787829997.239972:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829997.239985:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff9c1a7895e580 x1874668973235712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829997.239999:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973235712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829997.240006:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829997.240009:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829997.240013:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973235712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787829997.240035:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973235840:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829997.240054:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829997.240063:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff9c1a7895c000 x1874668973235840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829997.240073:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973235840/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829997.240081:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829997.240084:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829997.240090:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973235840:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829997.240100:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973235968:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787829997.240123:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787829997.240136:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829997/real 1787829997] req@ffff9c1a7895f480 x1874668973235968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829997.240146:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973235968/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829997.240156:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829997.240159:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787829997.240164:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973235968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787829997.751435:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829997.751435:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787829998.007598:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829998.775446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787829998.775446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787829999.031505:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829999.799443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787829999.799468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830000.055621:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830000.823454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830000.823468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830001.079520:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830001.847455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830001.847456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830002.103554:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830002.359459:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830002.359468:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830002.359472:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830002.359481:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6452 00000100:00080000:3.0:1787830002.359491:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830002.359497:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830002.359535:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973236096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830002.359561:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830002.359564:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830002.359566:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830002.359570:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6452 00000100:00080000:3.0:1787830002.359574:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830002.359578:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830002.359588:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830002.359603:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973236224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830002.359611:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973236096:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787830002.359616:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830002.359621:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830002.359622:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830002.359626:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6452 00000100:00080000:3.0:1787830002.359630:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830002.359633:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830002.359638:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830002.359651:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973236352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830002.359661:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830002.359665:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830002.359667:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830002.359670:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6452 00000800:00000100:1.0:1787830002.359674:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830002.359677:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830002.359683:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830002.359688:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830002.359699:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973236480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830002.359709:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830002.359713:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787830002.359715:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff9c1a7895d2c0 x1874668973236096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830002.359717:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830002.359724:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6452 00000100:00080000:3.0:1787830002.359732:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830002.359737:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830002.359741:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830002.359744:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973236096/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830002.359752:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973236608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830002.359759:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830002.359760:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830002.359766:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830002.359767:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830002.359769:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787830002.359772:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6452 00000100:00080000:3.0:1787830002.359775:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830002.359777:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973236096:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787830002.359780:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830002.359787:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830002.359794:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973236736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830002.359799:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830002.359801:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830002.359803:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830002.359805:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6452 00000100:00080000:3.0:1787830002.359808:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830002.359809:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973236224:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830002.359811:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830002.359817:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830002.359826:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973236864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830002.359854:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830002.359870:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff9c1a7895de00 x1874668973236224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830002.359885:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973236224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830002.359893:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830002.359896:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830002.359899:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973236224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830002.359905:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973236352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830002.359923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830002.359934:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff9c1a7895f840 x1874668973236352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830002.359944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973236352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830002.359951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830002.359954:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830002.359957:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973236352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830002.359963:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973236480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830002.359980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830002.359991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff9c1a7895f0c0 x1874668973236480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830002.360001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973236480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830002.360009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830002.360012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830002.360015:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973236480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830002.360020:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973236608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830002.360038:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830002.360049:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff9c1a7895fc00 x1874668973236608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830002.360060:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973236608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830002.360068:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830002.360071:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830002.360074:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973236608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830002.360079:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973236736:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830002.360097:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830002.360108:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff9c1a7895cb40 x1874668973236736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830002.360125:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973236736/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830002.360132:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830002.360135:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830002.360138:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973236736:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830002.360155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973236864:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830002.360173:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830002.360183:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830002/real 1787830002] req@ffff9c1a7895e1c0 x1874668973236864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830002.360194:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973236864/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830002.360202:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830002.360205:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830002.360208:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973236864:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830002.871467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830002.871909:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830003.127538:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830003.895462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830003.897183:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830004.151519:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830004.919725:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830004.919786:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830005.175633:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830005.943451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830005.943451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830006.199683:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830006.967462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830006.967462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830007.223989:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830007.479854:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830007.479865:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830007.479870:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830007.479880:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6457 00000100:00080000:3.0:1787830007.480090:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830007.480099:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830007.480137:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973236992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830007.480168:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830007.480171:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830007.480173:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830007.480177:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6457 00000100:00080000:3.0:1787830007.480181:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830007.480184:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830007.480194:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830007.480207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973237120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830007.480216:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830007.480219:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830007.480221:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830007.480225:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6457 00000100:00080000:3.0:1787830007.480227:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830007.480231:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830007.480234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830007.480243:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973237248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830007.480252:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830007.480255:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830007.480256:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830007.480260:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6457 00000100:00080000:3.0:1787830007.480262:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830007.480265:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830007.480269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830007.480277:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973237376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830007.480285:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830007.480288:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830007.480290:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830007.480293:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6457 00000100:00080000:3.0:1787830007.480295:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830007.480298:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830007.480302:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830007.480310:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973237504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830007.480317:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830007.480320:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830007.480322:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830007.480325:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6457 00000100:00080000:3.0:1787830007.480327:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830007.480330:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830007.480335:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830007.480460:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973236992:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830007.480518:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830007.480557:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830007/real 1787830007] req@ffff9c1a7895da40 x1874668973236992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830007.480573:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973236992/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830007.480582:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830007.480586:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830007.480590:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973236992:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830007.480597:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973237120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830007.480620:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830007.480631:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830007/real 1787830007] req@ffff9c1a7895e1c0 x1874668973237120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830007.480643:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973237120/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830007.480650:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830007.480652:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830007.480655:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973237120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830007.480660:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973237248:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830007.480688:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830007.480699:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830007/real 1787830007] req@ffff9c1a7895cb40 x1874668973237248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830007.480714:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973237248/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830007.480721:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830007.480724:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830007.480726:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973237248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830007.480731:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973237376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830007.480748:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830007.480758:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830007/real 1787830007] req@ffff9c1a7895fc00 x1874668973237376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830007.480768:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973237376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830007.480775:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830007.480778:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830007.480780:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973237376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830007.480786:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973237504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830007.480815:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830007.480826:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830007/real 1787830007] req@ffff9c1a7895f0c0 x1874668973237504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830007.480836:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973237504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830007.480847:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830007.480851:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830007.480853:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973237504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830007.481673:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973237632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830007.481699:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830007.481703:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830007.481705:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830007.481710:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6457 00000100:00080000:3.0:1787830007.481714:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830007.481719:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830007.481726:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830007.481743:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973237760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830007.482093:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973237632:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830007.482125:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830007.482140:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830007/real 1787830007] req@ffff9c1a7895f840 x1874668973237632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830007.482152:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973237632/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830007.482160:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830007.482164:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830007.482168:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973237632:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830007.482174:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973237760:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830007.482191:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830007.482202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830007/real 1787830007] req@ffff9c1a7895de00 x1874668973237760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830007.482213:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973237760/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830007.482221:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830007.482224:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830007.482227:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973237760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830007.991439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830007.991988:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830008.247807:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830009.015465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830009.015465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830009.271897:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830010.039430:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830010.039468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830010.295498:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830011.063419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830011.064585:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830011.320686:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830012.087461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830012.088101:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830012.344489:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830012.599428:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830012.599435:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830012.599438:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830012.599445:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6462 00000100:00080000:3.0:1787830012.599454:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830012.599459:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830012.599486:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973237888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830012.599509:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830012.599511:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830012.599512:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830012.599515:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6462 00000100:00080000:3.0:1787830012.599517:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830012.599519:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830012.599525:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830012.599532:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973238016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830012.599538:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830012.599540:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830012.599541:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830012.599543:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6462 00000100:00080000:3.0:1787830012.599545:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830012.599547:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830012.599549:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830012.599561:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973238144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830012.599568:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830012.599569:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830012.599571:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830012.599573:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6462 00000100:00080000:3.0:1787830012.599575:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830012.599577:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830012.599580:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830012.599590:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973238272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830012.599596:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830012.599598:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830012.599600:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830012.599602:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6462 00000100:00080000:3.0:1787830012.599604:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830012.599607:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830012.599612:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830012.599616:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973237888:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830012.599624:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973238400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830012.599636:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830012.599639:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830012.599641:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830012.599644:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6462 00000100:00080000:3.0:1787830012.599649:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830012.599653:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830012.599658:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830012.599668:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973238528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830012.599678:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830012.599679:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830012.599683:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830012.599685:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830012.599689:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6462 00000100:00080000:3.0:1787830012.599692:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830012.599697:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830012.599702:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830012.599713:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973238656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830012.599732:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830012/real 1787830012] req@ffff9c1a7895d2c0 x1874668973237888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830012.599820:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973237888/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830012.599831:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830012.599835:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830012.599838:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973237888:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830012.599845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973238016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830012.599868:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830012.599880:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830012/real 1787830012] req@ffff9c1a7895f480 x1874668973238016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830012.599891:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973238016/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830012.599919:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830012.599922:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830012.599926:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973238016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830012.599932:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973238144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830012.599957:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830012.599968:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830012/real 1787830012] req@ffff9c1a7895c000 x1874668973238144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830012.599979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973238144/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830012.599986:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830012.599989:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830012.599992:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973238144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830012.599997:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973238272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830012.600015:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830012.600027:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830012/real 1787830012] req@ffff9c1a7895e580 x1874668973238272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830012.600156:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973238272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830012.600165:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830012.600168:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830012.600171:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973238272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830012.600198:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973238400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830012.600230:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830012.600243:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830012/real 1787830012] req@ffff9c1a7895cf00 x1874668973238400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830012.600255:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973238400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830012.600263:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830012.600265:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830012.600268:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973238400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830012.600274:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973238528:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830012.600294:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830012.600305:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830012/real 1787830012] req@ffff9c1a7895e940 x1874668973238528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830012.600317:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973238528/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830012.600325:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830012.600328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830012.600331:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973238528:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830012.600546:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973238656:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830012.600570:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830012.600850:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830012/real 1787830012] req@ffff9c1a7895c780 x1874668973238656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830012.600864:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973238656/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830012.600871:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830012.600874:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830012.600877:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973238656:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830013.111467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830013.111620:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830013.367613:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830014.135531:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830014.135779:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830014.391525:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830015.159451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830015.159606:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830015.415564:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830016.183470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830016.185788:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830016.440067:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830017.207455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830017.207481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830017.463572:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830017.719773:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830017.719783:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830017.719788:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830017.719797:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6468 00000100:00080000:1.0:1787830017.719808:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830017.719816:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830017.719866:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973238784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830017.719893:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830017.719897:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830017.719899:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830017.719903:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6468 00000100:00080000:1.0:1787830017.719907:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830017.719911:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830017.719920:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830017.719934:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973238784:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830017.719945:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973238912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830017.719957:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830017.719961:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830017.719963:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830017.719967:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6468 00000100:00080000:1.0:1787830017.719971:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830017.719974:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830017.719979:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830017.719990:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973239040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830017.719999:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830017.720002:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830017.720004:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830017.720007:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6468 00000100:00080000:1.0:1787830017.720010:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830017.720014:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830017.720018:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830017.720023:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830017.720030:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973239168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830017.720046:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830017.720050:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830017.720052:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830017.720056:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6468 00000100:00080000:1.0:1787830017.720060:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830017.720063:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830017/real 1787830017] req@ffff9c1a78a7b0c0 x1874668973238784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830017.720078:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973238784/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830017.720079:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830017.720089:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787830017.720090:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830017.720096:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830017.720099:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973238784:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830017.720107:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973239296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830017.720119:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973238912:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830017.720120:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830017.720127:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830017.720129:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830017.720144:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6468 00000800:00000100:2.0:1787830017.720147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830017.720152:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830017.720158:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830017/real 1787830017] req@ffff9c1a78a7a1c0 x1874668973238912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830017.720167:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830017.720172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973238912/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787830017.720181:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787830017.720181:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830017.720187:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830017.720190:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973238912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830017.720195:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973239040:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830017.720200:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973239424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830017.720209:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830017.720214:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830017.720219:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830017/real 1787830017] req@ffff9c1a78a783c0 x1874668973239040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830017.720220:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830017.720224:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830017.720234:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6468 00000100:00100000:2.0:1787830017.720235:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973239040/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830017.720242:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830017.720244:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830017.720249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787830017.720252:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830017.720253:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973239040:192.168.203.148@tcp:38:kworker.0 00000008:00100000:1.0:1787830017.720262:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830017.720262:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973239168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830017.720281:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973239552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830017.720282:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830017.720290:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830017/real 1787830017] req@ffff9c1a78a79e00 x1874668973239168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830017.720302:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973239168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830017.720307:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830017.720309:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830017.720310:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973239168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830017.720324:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973239296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830017.720333:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830017.720371:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830017/real 1787830017] req@ffff9c1a78a7a940 x1874668973239296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830017.720382:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973239296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830017.720390:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830017.720393:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830017.720396:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973239296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830017.720407:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973239424:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830017.720425:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830017.720446:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830017/real 1787830017] req@ffff9c1a78a78780 x1874668973239424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830017.720457:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973239424/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830017.720464:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830017.720467:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830017.720470:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973239424:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830017.720475:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973239552:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830017.720492:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830017.720502:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830017/real 1787830017] req@ffff9c1a78a7bc00 x1874668973239552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830017.720513:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973239552/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830017.720527:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830017.720530:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830017.720533:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973239552:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830018.231449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830018.231472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830018.487954:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830019.255453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830019.255456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830019.511670:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830020.279460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830020.279460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830020.535527:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830021.303440:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830021.303440:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830021.559568:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830022.327614:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830022.327649:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830022.585320:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830022.839519:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830022.839529:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830022.839533:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830022.839543:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6473 00000100:00080000:2.0:1787830022.839553:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830022.839559:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830022.839596:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973239680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830022.839655:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973239680:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830022.839717:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787830022.839741:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830022.839744:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830022.839745:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830022.839748:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6473 00000100:00080000:2.0:1787830022.839751:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830022.839754:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830022.839761:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830022.839770:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973239808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830022.839822:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830022/real 1787830022] req@ffff9c1a49d8cb40 x1874668973239680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830022.839838:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973239680/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830022.839848:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830022.839851:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830022.839855:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973239680:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830022.839875:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973239808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830022.839898:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830022.839910:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830022/real 1787830022] req@ffff9c1a49d8da40 x1874668973239808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830022.839922:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973239808/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830022.839929:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830022.839932:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830022.839935:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973239808:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787830022.839989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830022.839993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830022.839996:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830022.840000:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6473 00000100:00080000:2.0:1787830022.840004:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830022.840008:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830022.840014:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830022.840026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973239936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830022.840038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830022.840041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830022.840043:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830022.840046:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6473 00000100:00080000:2.0:1787830022.840049:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830022.840053:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830022.840057:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830022.840067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973240064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830022.840075:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830022.840078:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830022.840080:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830022.840083:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6473 00000100:00080000:2.0:1787830022.840086:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830022.840089:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830022.840093:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830022.840102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973240192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830022.840110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830022.840112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830022.840114:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830022.840117:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6473 00000100:00080000:2.0:1787830022.840119:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830022.840123:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830022.840127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830022.840136:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973240320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830022.840144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830022.840147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830022.840148:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830022.840151:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6473 00000100:00080000:2.0:1787830022.840154:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830022.840157:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830022.840161:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830022.840170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973240448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830022.840228:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973239936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830022.840252:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830022.840263:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830022/real 1787830022] req@ffff9c1a49d8de00 x1874668973239936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830022.840274:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973239936/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830022.840295:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830022.840298:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830022.840301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973239936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830022.840311:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973240064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830022.840328:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830022.840337:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830022/real 1787830022] req@ffff9c1a49d8c780 x1874668973240064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830022.840365:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973240064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830022.840379:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830022.840382:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830022.840385:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973240064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830022.840390:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973240192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830022.840408:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830022.840608:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830022/real 1787830022] req@ffff9c1a49d8c000 x1874668973240192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830022.840621:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973240192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830022.840628:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830022.840631:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830022.840634:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973240192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830022.840639:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973240320:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830022.840660:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830022.840670:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830022/real 1787830022] req@ffff9c1a49d8cf00 x1874668973240320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830022.840680:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973240320/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830022.840688:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830022.840691:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830022.840708:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973240320:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830022.840714:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973240448:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830022.840733:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830022.840743:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830022/real 1787830022] req@ffff9c1a49d8da40 x1874668973240448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830022.840753:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973240448/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830022.840760:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830022.840763:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830022.840766:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973240448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830023.351448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830023.351495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830023.607630:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830024.375475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830024.376180:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830024.633791:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830025.399459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830025.399858:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830025.655609:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830026.423455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830026.423455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830026.680516:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830027.447475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830027.447592:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830027.706777:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830027.959504:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830027.959515:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830027.959519:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830027.959528:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6478 00000100:00080000:0.0:1787830027.959538:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830027.959544:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830027.959581:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973240576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830027.959609:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830027.959613:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830027.959615:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830027.959619:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6478 00000100:00080000:0.0:1787830027.959623:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830027.959627:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830027.959636:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830027.959647:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973240704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830027.959656:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830027.959672:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830027.959674:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830027.959679:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6478 00000100:00080000:0.0:1787830027.959683:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830027.959686:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830027.959691:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830027.959702:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973240832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830027.959712:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830027.959715:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830027.959717:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830027.959720:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6478 00000100:00080000:0.0:1787830027.959724:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830027.959727:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830027.959731:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830027.959739:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973240960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830027.959747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830027.959750:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830027.959752:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830027.959755:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6478 00000100:00080000:0.0:1787830027.959757:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830027.959760:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830027.959764:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830027.959771:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973241088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830027.959780:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830027.959782:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830027.959784:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830027.959787:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6478 00000100:00080000:0.0:1787830027.959789:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830027.959792:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830027.959796:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830027.959805:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973241216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830027.959813:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830027.959816:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830027.959817:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830027.959821:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6478 00000100:00080000:0.0:1787830027.959824:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830027.959827:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830027.959831:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830027.959839:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973241344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830027.959983:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973240576:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830027.960037:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830027.960070:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830027/real 1787830027] req@ffff9c1a78860780 x1874668973240576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830027.960086:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973240576/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830027.960101:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830027.960104:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830027.960108:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973240576:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830027.960114:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973240704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830027.960131:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830027.960141:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830027/real 1787830027] req@ffff9c1a78861a40 x1874668973240704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830027.960151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973240704/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830027.960158:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830027.960161:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830027.960164:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973240704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830027.960169:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973240832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830027.960183:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830027.960191:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830027/real 1787830027] req@ffff9c1a78862940 x1874668973240832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830027.960208:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973240832/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830027.960215:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830027.960219:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830027.960222:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973240832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830027.960228:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973240960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830027.960245:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830027.960256:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830027/real 1787830027] req@ffff9c1a78860b40 x1874668973240960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830027.960408:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973240960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830027.960418:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830027.960422:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830027.960425:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973240960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830027.960431:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973241088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830027.960453:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830027.960465:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830027/real 1787830027] req@ffff9c1a788621c0 x1874668973241088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830027.960485:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973241088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830027.960496:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830027.960499:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830027.960502:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973241088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830027.960507:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973241216:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830027.960525:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830027.960536:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830027/real 1787830027] req@ffff9c1a78863840 x1874668973241216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830027.960547:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973241216/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830027.960552:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830027.960554:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830027.960556:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973241216:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830027.960559:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973241344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830027.960572:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830027.960578:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830027/real 1787830027] req@ffff9c1a78862d00 x1874668973241344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830027.960585:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973241344/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830027.960589:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830027.960591:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830027.960593:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973241344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830028.471796:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830028.471804:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830028.727561:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830029.495523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830029.495714:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830029.751508:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830030.519453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830030.519466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830030.776173:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830031.543506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830031.543506:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830031.799642:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830032.567469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830032.567475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830032.823583:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830033.079576:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830033.079689:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830033.079695:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830033.079704:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6483 00000100:00080000:0.0:1787830033.079715:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830033.079723:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830033.079755:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973241472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830033.079789:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830033.079793:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830033.079795:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830033.079799:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6483 00000100:00080000:0.0:1787830033.079802:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830033.079805:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830033.079814:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830033.079827:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973241600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830033.079839:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830033.079842:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830033.079844:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830033.079848:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973241472:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787830033.079850:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6483 00000100:00080000:0.0:1787830033.079858:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830033.079863:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830033.079868:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830033.079879:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973241728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830033.079890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830033.079893:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830033.079895:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830033.079899:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6483 00000100:00080000:0.0:1787830033.079902:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830033.079907:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830033.079911:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830033.079922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973241856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830033.079932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830033.079935:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830033.079937:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830033.079941:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6483 00000100:00080000:0.0:1787830033.079945:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830033.079949:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830033.079955:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830033.079955:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973241600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830033.079971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973241984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830033.079981:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973241728:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830033.079983:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830033.079989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830033.079991:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830033.079996:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6483 00000100:00100000:3.0:1787830033.079999:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973241856:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830033.080002:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830033.080014:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973241984:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830033.080064:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830033.080070:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830033.080082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973242112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830033.080098:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830033.080101:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830033.080103:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830033.080106:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6483 00000100:00080000:0.0:1787830033.080109:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830033.080113:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830033.080117:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830033.080128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973242240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830033.080178:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973242112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830033.080207:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973242240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830033.591496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830033.591496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830033.848137:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830034.615545:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830034.615682:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830034.871598:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830035.639492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830035.640511:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830035.895594:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787830036.151626:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:3.0:1787830036.151674:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830036.151692:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787830036.152042:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830036] req@ffff9c1a78860f00 x1874668973241472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830058 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.152072:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973241472/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830058 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.152087:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.152092:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830036.152097:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973241472:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830036.154408:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787830036.154452:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787830036.154462:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787830036.154469:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787830036.154476:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:3.0:1787830036.154483:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:0.0:1787830036.154881:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830036] req@ffff9c1a78862940 x1874668973242112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.154904:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973242112/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.154916:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.154921:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830036.154926:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973242112:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830036.154934:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830036] req@ffff9c1a78861a40 x1874668973242240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.154946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973242240/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.154954:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.154957:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830036.154961:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973242240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830036.154967:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830036] req@ffff9c1a78860b40 x1874668973241984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.154977:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973241984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.154995:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.154997:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830036.154999:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973241984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830036.155004:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830036] req@ffff9c1a78862d00 x1874668973241600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.155011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973241600/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.155016:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.155018:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830036.155020:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973241600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830036.155023:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830036] req@ffff9c1a78863840 x1874668973241728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.155029:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973241728/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.155035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.155038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830036.155041:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973241728:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830036.155047:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830033/real 1787830036] req@ffff9c1a788621c0 x1874668973241856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.155057:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973241856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.155065:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.155068:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830036.155071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973241856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830036.663480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830036.663496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830036.919922:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830037.687471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830037.687489:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830037.943832:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830038.200603:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830038.200613:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830038.200617:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830038.200627:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6488 00000100:00080000:0.0:1787830038.200636:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830038.200643:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830038.200682:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973242368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830038.200710:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830038.200713:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830038.200715:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830038.200719:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6488 00000100:00080000:0.0:1787830038.200722:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830038.200726:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830038.200734:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830038.200746:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973242496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830038.200754:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830038.200756:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830038.200758:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830038.200761:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6488 00000100:00080000:0.0:1787830038.200763:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830038.200766:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830038.200769:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830038.200776:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973242624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830038.200798:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830038.200801:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830038.200803:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830038.200806:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6488 00000100:00080000:0.0:1787830038.200809:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830038.200812:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830038.200816:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830038.200825:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973242752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830038.200833:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830038.200835:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830038.200837:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830038.200841:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6488 00000100:00080000:0.0:1787830038.200843:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830038.200847:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830038.200851:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830038.200860:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973242880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830038.200868:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830038.200870:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830038.200872:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830038.200875:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6488 00000100:00080000:0.0:1787830038.200878:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830038.200881:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830038.200886:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830038.200894:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973243008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830038.200901:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830038.200904:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830038.200906:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830038.200909:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6488 00000100:00080000:0.0:1787830038.200912:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830038.200915:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830038.200918:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830038.200925:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973243136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830038.201894:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973242368:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830038.201990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830038.202042:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830038/real 1787830038] req@ffff9c1a788621c0 x1874668973242368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830038.202062:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973242368/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830038.202075:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830038.202080:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830038.202085:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973242368:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830038.202094:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973242496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830038.202118:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830038.202131:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830038/real 1787830038] req@ffff9c1a78863840 x1874668973242496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830038.202144:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973242496/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830038.202152:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830038.202156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830038.202159:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973242496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830038.202166:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973242624:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830038.202185:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830038.202197:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830038/real 1787830038] req@ffff9c1a78862d00 x1874668973242624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830038.202208:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973242624/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830038.203388:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830038.203395:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830038.203400:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973242624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830038.203412:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973242752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830038.203468:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830038.203486:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830038/real 1787830038] req@ffff9c1a78860b40 x1874668973242752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830038.203500:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973242752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830038.203636:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830038.203641:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830038.203645:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973242752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830038.203654:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973242880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830038.203680:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830038.203705:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830038/real 1787830038] req@ffff9c1a78861a40 x1874668973242880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830038.203719:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973242880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830038.203728:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830038.203732:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830038.203736:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973242880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830038.203754:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973243008:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830038.203777:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830038.203789:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830038/real 1787830038] req@ffff9c1a78862940 x1874668973243008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830038.203801:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973243008/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830038.203809:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830038.203812:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830038.203815:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973243008:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830038.203822:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973243136:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830038.203843:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830038.203853:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830038/real 1787830038] req@ffff9c1a78860780 x1874668973243136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830038.203878:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973243136/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830038.203885:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830038.203888:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830038.203891:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973243136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830038.711467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830038.711850:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830038.967537:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830039.735480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830039.735823:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830039.992627:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830040.759557:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830040.759671:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830041.015867:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830041.783448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830041.783755:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830042.039732:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830042.807480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830042.807491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830043.063600:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830043.320013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830043.320023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830043.320027:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830043.320048:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6493 00000100:00080000:2.0:1787830043.320059:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830043.320067:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830043.320098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973243264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830043.320130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830043.320134:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830043.320135:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830043.320140:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6493 00000100:00080000:2.0:1787830043.320144:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830043.320148:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830043.320156:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830043.320168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973243392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830043.320177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830043.320179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830043.320181:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830043.320184:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6493 00000100:00080000:2.0:1787830043.320187:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830043.320191:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830043.320195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830043.320203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973243520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830043.320218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830043.320221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830043.320223:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830043.320226:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6493 00000100:00080000:2.0:1787830043.320229:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830043.320232:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830043.320236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830043.320243:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973243648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830043.320251:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830043.320254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830043.320256:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830043.320259:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6493 00000100:00080000:2.0:1787830043.320261:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830043.320275:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830043.320279:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830043.320289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973243776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830043.320297:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830043.320300:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830043.320302:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830043.320305:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6493 00000100:00080000:2.0:1787830043.320309:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830043.320312:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830043.320316:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830043.320324:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973243904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830043.320332:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830043.320335:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830043.320337:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830043.320340:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6493 00000100:00080000:2.0:1787830043.320342:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830043.320370:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830043.320375:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830043.320386:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973244032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830043.320472:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973243264:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830043.320538:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830043.320589:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830043/real 1787830043] req@ffff9c1a49d8cb40 x1874668973243264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830043.320606:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973243264/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830043.320615:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830043.320619:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830043.320623:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973243264:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830043.320630:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973243392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830043.320653:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830043.320666:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830043/real 1787830043] req@ffff9c1a49d8e1c0 x1874668973243392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830043.320677:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973243392/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830043.320684:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830043.320688:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830043.320690:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973243392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830043.320699:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973243520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830043.320717:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830043.320727:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830043/real 1787830043] req@ffff9c1a49d8fc00 x1874668973243520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830043.320738:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973243520/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830043.320745:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830043.320747:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830043.320750:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973243520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830043.320755:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973243648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830043.320771:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830043.320781:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830043/real 1787830043] req@ffff9c1a49d8ed00 x1874668973243648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830043.320789:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973243648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830043.320796:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830043.320799:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830043.320801:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973243648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830043.320806:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973243776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830043.320820:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830043.320828:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830043/real 1787830043] req@ffff9c1a49d8f0c0 x1874668973243776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830043.320839:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973243776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830043.320846:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830043.320849:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830043.320852:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973243776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830043.320857:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973243904:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830043.320873:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830043.320883:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830043/real 1787830043] req@ffff9c1a49d8c3c0 x1874668973243904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830043.320892:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973243904/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830043.320899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830043.320902:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830043.320904:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973243904:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830043.320910:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973244032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830043.320926:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830043.320935:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830043/real 1787830043] req@ffff9c1a49d8e580 x1874668973244032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830043.320944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973244032/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830043.320957:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830043.320960:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830043.320963:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973244032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830043.831468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830043.841666:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830044.087527:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830044.855504:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830044.855514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830045.113586:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830045.879412:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830045.879507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830046.136220:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830046.903561:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830046.903735:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830047.159714:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830047.927453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830047.927453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830048.183647:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830048.439481:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830048.439491:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830048.439496:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830048.439505:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6498 00000100:00080000:0.0:1787830048.439529:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830048.439538:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830048.439574:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973244160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830048.439602:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830048.439606:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830048.439608:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830048.439612:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6498 00000100:00080000:0.0:1787830048.439615:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830048.439619:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830048.439628:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830048.439640:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973244288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830048.439652:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830048.439656:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830048.439658:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830048.439662:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973244160:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787830048.439664:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6498 00000100:00080000:0.0:1787830048.439670:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830048.439675:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830048.439680:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830048.439691:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973244416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830048.439702:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830048.439705:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830048.439707:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830048.439710:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6498 00000100:00080000:0.0:1787830048.439715:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830048.439719:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830048.439724:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787830048.439727:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830048.439738:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973244544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830048.439747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830048.439750:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830048.439751:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830048.439754:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6498 00000100:00080000:0.0:1787830048.439758:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830048.439762:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830048.439767:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830048.439770:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830048/real 1787830048] req@ffff9c1a78860000 x1874668973244160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830048.439780:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973244672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830048.439790:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830048.439793:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973244160/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830048.439795:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830048.439799:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830048.439803:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6498 00000100:00080000:0.0:1787830048.439806:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830048.439809:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830048.439814:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830048.439824:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973244800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830048.439832:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830048.439835:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830048.439836:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830048.439839:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6498 00000100:00080000:0.0:1787830048.439841:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830048.439844:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830048.439848:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830048.439856:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973244928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830048.439921:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830048.439925:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830048.439929:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973244160:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830048.439954:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973244288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830048.439977:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830048.440001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830048/real 1787830048] req@ffff9c1a788630c0 x1874668973244288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830048.440020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973244288/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830048.440028:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830048.440030:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830048.440034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973244288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830048.440039:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973244416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830048.440057:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830048.440068:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830048/real 1787830048] req@ffff9c1a78862580 x1874668973244416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830048.440079:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973244416/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830048.440086:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830048.440089:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830048.440092:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973244416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830048.440098:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973244544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830048.440116:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830048.440127:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830048/real 1787830048] req@ffff9c1a78861680 x1874668973244544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830048.440137:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973244544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830048.440151:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830048.440155:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830048.440158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973244544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830048.440163:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973244672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830048.440180:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830048.440191:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830048/real 1787830048] req@ffff9c1a78861e00 x1874668973244672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830048.440202:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973244672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830048.440210:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830048.440213:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830048.440216:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973244672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830048.440221:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973244800:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830048.440240:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830048.440252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830048/real 1787830048] req@ffff9c1a788612c0 x1874668973244800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830048.440263:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973244800/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830048.440270:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830048.440273:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830048.440277:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973244800:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830048.440282:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973244928:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830048.440300:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830048.440310:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830048/real 1787830048] req@ffff9c1a78863c00 x1874668973244928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830048.440321:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973244928/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830048.440329:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830048.440332:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830048.440335:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973244928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830048.951509:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830048.951510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830049.208138:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830049.975490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830049.975621:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830050.231715:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830050.999447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830050.999447:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830051.255568:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830052.023460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830052.023469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830052.279625:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830053.047458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830053.047530:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830053.303535:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830053.559494:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830053.559504:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830053.559509:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830053.559518:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6503 00000100:00080000:0.0:1787830053.559528:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830053.559534:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830053.559576:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973245056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830053.559616:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830053.559620:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830053.559622:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830053.559626:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6503 00000100:00080000:0.0:1787830053.559629:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830053.559633:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830053.559641:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830053.559653:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973245184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830053.559662:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830053.559665:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830053.559667:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830053.559670:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6503 00000100:00080000:0.0:1787830053.559673:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830053.559684:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830053.559688:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830053.559696:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973245312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830053.559706:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830053.559709:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830053.559710:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830053.559714:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6503 00000100:00080000:0.0:1787830053.559718:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830053.559721:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830053.559725:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830053.559733:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973245440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830053.559740:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830053.559743:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830053.559745:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830053.559748:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6503 00000100:00080000:0.0:1787830053.559750:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830053.559753:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830053.559763:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830053.559771:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973245568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830053.559779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830053.559782:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830053.559784:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830053.559787:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6503 00000100:00080000:0.0:1787830053.559790:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830053.559793:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830053.559797:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830053.559804:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973245696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830053.559824:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830053.559829:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830053.559831:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830053.559834:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6503 00000100:00080000:0.0:1787830053.559837:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830053.559840:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830053.559843:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830053.559851:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973245824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830053.559943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973245056:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830053.559996:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830053.560031:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830053/real 1787830053] req@ffff9c1a78863c00 x1874668973245056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830053.560047:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973245056/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830053.560058:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830053.560061:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830053.560065:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973245056:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830053.560071:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973245184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830053.560090:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830053.560101:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830053/real 1787830053] req@ffff9c1a788612c0 x1874668973245184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830053.560112:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973245184/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830053.560121:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830053.560124:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830053.560127:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973245184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830053.560133:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973245312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830053.560148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830053.560165:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830053/real 1787830053] req@ffff9c1a78861e00 x1874668973245312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830053.560176:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973245312/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830053.560182:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830053.560185:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830053.560188:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973245312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830053.560193:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973245440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830053.560211:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830053.560221:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830053/real 1787830053] req@ffff9c1a78861680 x1874668973245440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830053.560232:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973245440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830053.560241:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830053.560243:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830053.560246:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973245440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830053.560251:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973245568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830053.560278:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830053.560289:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830053/real 1787830053] req@ffff9c1a78862580 x1874668973245568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830053.560299:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973245568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830053.560306:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830053.560309:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830053.560312:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973245568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830053.560317:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973245696:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830053.560335:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830053.560368:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830053/real 1787830053] req@ffff9c1a788630c0 x1874668973245696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830053.560380:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973245696/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830053.560388:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830053.560390:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830053.560393:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973245696:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830053.560399:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973245824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830053.560418:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830053.560429:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830053/real 1787830053] req@ffff9c1a78860000 x1874668973245824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830053.560440:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973245824/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830053.560447:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830053.560449:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830053.560452:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973245824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830054.071441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830054.071441:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830054.327525:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830055.095485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830055.095765:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830055.353710:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830056.119452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830056.119566:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830056.375885:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830057.143681:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830057.143980:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830057.399616:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830058.167477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830058.167493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830058.425341:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830058.680499:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830058.680509:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830058.680514:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830058.680632:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6509 00000100:00080000:3.0:1787830058.680642:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830058.680649:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830058.680690:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973245952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830058.680721:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830058.680725:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830058.680727:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830058.680731:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6509 00000100:00080000:3.0:1787830058.680735:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830058.680739:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830058.680747:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830058.680761:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973246080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830058.680769:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830058.680772:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830058.680774:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830058.680777:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6509 00000100:00080000:3.0:1787830058.680781:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830058.680784:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830058.680788:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830058.680796:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973246208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830058.680806:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830058.680809:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830058.680811:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830058.680814:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6509 00000100:00080000:3.0:1787830058.680817:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830058.680820:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830058.680824:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830058.680833:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973246336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830058.680842:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830058.680845:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830058.680847:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830058.680850:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6509 00000100:00080000:3.0:1787830058.680853:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830058.680856:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830058.680860:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830058.680869:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973246464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830058.680877:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830058.680880:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830058.680882:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830058.680884:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6509 00000100:00080000:3.0:1787830058.680888:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830058.680891:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830058.680896:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830058.680905:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973246592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830058.680912:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830058.680915:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830058.680916:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830058.680919:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6509 00000100:00080000:3.0:1787830058.680922:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830058.680925:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830058.680929:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830058.680938:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973246720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830058.681021:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973245952:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787830058.681074:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830058.681109:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830058/real 1787830058] req@ffff9c1a7895c3c0 x1874668973245952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830058.681125:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973245952/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830058.681135:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830058.681138:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830058.681141:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973245952:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830058.681148:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973246080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830058.681169:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830058.681181:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830058/real 1787830058] req@ffff9c1a7895c780 x1874668973246080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830058.681193:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973246080/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830058.681200:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830058.681203:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830058.681206:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973246080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830058.681210:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973246208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830058.681227:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830058.681236:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830058/real 1787830058] req@ffff9c1a7895e940 x1874668973246208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830058.681246:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973246208/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830058.681252:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830058.681255:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830058.681258:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973246208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830058.681263:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973246336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830058.681277:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830058.681285:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830058/real 1787830058] req@ffff9c1a7895cf00 x1874668973246336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830058.681295:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973246336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830058.681307:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830058.681309:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830058.681312:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973246336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830058.681316:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973246464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830058.681330:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830058.681338:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830058/real 1787830058] req@ffff9c1a7895e580 x1874668973246464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830058.681369:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973246464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830058.681376:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830058.681378:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830058.681381:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973246464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830058.681387:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973246592:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830058.681406:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830058.681416:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830058/real 1787830058] req@ffff9c1a7895c000 x1874668973246592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830058.681427:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973246592/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830058.681434:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830058.681436:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830058.681439:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973246592:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830058.681457:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973246720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830058.681475:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830058.681486:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830058/real 1787830058] req@ffff9c1a7895f480 x1874668973246720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830058.681498:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973246720/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830058.681506:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830058.681508:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830058.681511:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973246720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830059.191497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830059.191595:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830059.447691:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830060.215502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830060.215677:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830060.471765:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830061.240107:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830061.242264:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830061.495669:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830062.263472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830062.263472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830062.519962:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830063.287577:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830063.287986:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830063.545677:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830063.799626:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830063.799636:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830063.799640:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830063.799649:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6514 00000100:00080000:2.0:1787830063.799659:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830063.799666:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830063.799700:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973246848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830063.799729:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830063.799733:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830063.799735:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830063.799739:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6514 00000100:00080000:2.0:1787830063.799743:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830063.799747:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830063.799756:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830063.799770:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973246976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830063.799781:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830063.799791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830063.799793:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830063.799797:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6514 00000100:00080000:2.0:1787830063.799801:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830063.799804:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830063.799809:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830063.799819:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973247104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830063.799829:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830063.799832:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830063.799833:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830063.799836:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6514 00000100:00080000:2.0:1787830063.799840:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830063.799843:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830063.799848:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830063.799857:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973247232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830063.799865:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830063.799868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830063.799870:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830063.799873:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6514 00000100:00080000:2.0:1787830063.799876:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830063.799880:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830063.799885:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830063.799895:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973247360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830063.799904:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830063.799907:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830063.799909:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830063.799912:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6514 00000100:00080000:2.0:1787830063.799915:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830063.799918:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830063.799924:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830063.799934:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973247488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830063.799942:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830063.799945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830063.799946:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830063.799949:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6514 00000100:00080000:2.0:1787830063.799952:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830063.799955:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830063.799959:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830063.799969:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973247616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830063.800048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973246848:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787830063.800104:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830063.800149:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830063/real 1787830063] req@ffff9c1a49d8d680 x1874668973246848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830063.800164:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973246848/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830063.800173:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830063.800176:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830063.800179:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973246848:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830063.800185:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973246976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830063.800213:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830063.800224:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830063/real 1787830063] req@ffff9c1a49d8e580 x1874668973246976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830063.800234:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973246976/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830063.800241:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830063.800244:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830063.800247:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973246976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830063.800251:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973247104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830063.800268:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830063.800278:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830063/real 1787830063] req@ffff9c1a49d8c3c0 x1874668973247104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830063.800289:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973247104/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830063.800296:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830063.800298:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830063.800301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973247104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830063.800306:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973247232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830063.800322:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830063.800330:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830063/real 1787830063] req@ffff9c1a49d8f0c0 x1874668973247232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830063.800340:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973247232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830063.800373:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830063.800376:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830063.800378:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973247232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830063.800383:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973247360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830063.800399:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830063.800421:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830063/real 1787830063] req@ffff9c1a49d8ed00 x1874668973247360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830063.800432:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973247360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830063.800439:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830063.800442:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830063.800444:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973247360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830063.800449:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973247488:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830063.800466:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830063.800474:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830063/real 1787830063] req@ffff9c1a49d8fc00 x1874668973247488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830063.800484:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973247488/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830063.800490:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830063.800493:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830063.800496:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973247488:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830063.800501:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973247616:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830063.800514:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830063.800522:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830063/real 1787830063] req@ffff9c1a49d8e1c0 x1874668973247616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830063.800532:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973247616/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830063.800539:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830063.800541:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830063.800544:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973247616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830064.311429:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830064.313204:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830064.568042:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830065.335467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830065.336048:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830065.592101:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830066.359472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830066.360033:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830066.615664:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830067.383480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830067.383609:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830067.639563:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830068.407632:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830068.408428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830068.664190:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830068.919853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830068.919873:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830068.919878:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830068.919892:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6519 00000100:00080000:0.0:1787830068.919909:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830068.919918:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830068.919971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973247744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830068.920014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830068.920018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830068.920021:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830068.920026:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6519 00000100:00080000:0.0:1787830068.920030:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830068.920037:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830068.920049:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830068.920063:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973247744:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830068.920065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973247872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830068.920099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830068.920103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830068.920105:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830068.920111:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6519 00000800:00000100:1.0:1787830068.920143:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830068.920181:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830068/real 1787830068] req@ffff9c1a78860780 x1874668973247744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830068.920208:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973247744/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830068.920220:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830068.920225:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830068.920231:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973247744:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830068.920263:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973247872:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830068.920624:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830068.920633:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830068.920641:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830068.920660:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973248000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830068.920676:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830068.920680:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830068.920684:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787830068.920690:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830068.920692:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6519 00000100:00080000:0.0:1787830068.920702:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830068.920710:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830068.920719:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830068/real 1787830068] req@ffff9c1a78862940 x1874668973247872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830068.920736:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973247872/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787830068.920737:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787830068.920756:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787830068.920760:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973248128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830068.920765:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830068.920780:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973247872:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830068.920787:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830068.920793:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830068.920796:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830068.920801:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6519 00000100:00100000:1.0:1787830068.920805:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973248000:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830068.920809:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830068.920817:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830068.920824:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787830068.920836:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830068.920844:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973248256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830068.920858:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830068.920861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830068.920863:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830068.920867:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6519 00000100:00080000:0.0:1787830068.920871:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830068.920874:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830068.920880:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830068.920891:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973248384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830068.920900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830068.920904:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830068.920906:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830068.920913:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830068/real 1787830068] req@ffff9c1a78861a40 x1874668973248000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830068.920914:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6519 00000100:00080000:0.0:1787830068.920930:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830068.920937:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830068.920940:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973248000/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787830068.920945:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787830068.920954:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830068.920959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830068.920963:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973248000:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830068.920972:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973248128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830068.920976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973248512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830068.921003:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830068.921017:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830068/real 1787830068] req@ffff9c1a78860b40 x1874668973248128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830068.921027:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973248128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830068.921034:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830068.921036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830068.921038:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973248128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830068.921053:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973248256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830068.921081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830068.921087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830068/real 1787830068] req@ffff9c1a78862d00 x1874668973248256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830068.921094:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973248256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830068.921099:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830068.921101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830068.921102:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973248256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830068.921106:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973248384:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830068.921115:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830068.921120:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830068/real 1787830068] req@ffff9c1a78863840 x1874668973248384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830068.921126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973248384/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830068.921130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830068.921132:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830068.921134:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973248384:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830068.921138:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973248512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830068.921147:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830068.921153:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830068/real 1787830068] req@ffff9c1a788621c0 x1874668973248512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830068.921163:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973248512/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830068.921168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830068.921170:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830068.921173:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973248512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830069.431482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830069.431585:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830069.687681:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830070.455501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830070.455608:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830070.711612:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830071.479569:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830071.541901:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830071.735723:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830072.503498:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830072.567473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830072.760132:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830073.527503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830073.605111:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830073.783654:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830074.040283:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830074.040292:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830074.040296:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830074.040305:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6524 00000100:00080000:3.0:1787830074.040314:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830074.040321:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830074.040384:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973248640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830074.040411:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830074.040415:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830074.040417:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830074.040421:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6524 00000100:00080000:3.0:1787830074.040424:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830074.040428:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830074.040437:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830074.040448:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973248768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830074.040457:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830074.040460:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830074.040461:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830074.040465:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6524 00000100:00080000:3.0:1787830074.040469:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830074.040472:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830074.040476:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830074.040483:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973248896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830074.040492:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830074.040495:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830074.040497:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830074.040500:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6524 00000100:00080000:3.0:1787830074.040503:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830074.040506:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830074.040510:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830074.040518:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973249024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830074.040527:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830074.040529:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830074.040531:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830074.040533:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6524 00000100:00080000:3.0:1787830074.040536:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830074.040539:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830074.040542:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830074.040550:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973249152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830074.040563:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830074.040565:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830074.040567:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830074.040570:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6524 00000100:00080000:3.0:1787830074.040572:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830074.040575:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830074.040579:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830074.040587:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973249280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830074.040594:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830074.040597:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830074.040599:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830074.040601:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6524 00000100:00080000:3.0:1787830074.040604:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830074.040607:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830074.040610:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830074.040617:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973249408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830074.040718:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973248640:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830074.040772:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830074.040806:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830074/real 1787830074] req@ffff9c1a7895d2c0 x1874668973248640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830074.040820:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973248640/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830074.040841:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830074.040846:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830074.040849:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973248640:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830074.040856:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973248768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830074.040885:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830074.040896:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830074/real 1787830074] req@ffff9c1a7895de00 x1874668973248768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830074.040907:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973248768/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830074.040914:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830074.040917:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830074.040920:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973248768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830074.040926:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973248896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830074.040941:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830074.040949:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830074/real 1787830074] req@ffff9c1a7895f840 x1874668973248896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830074.040960:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973248896/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830074.040967:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830074.040970:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830074.040973:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973248896:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830074.040978:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973249024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830074.040992:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830074.041001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830074/real 1787830074] req@ffff9c1a7895f0c0 x1874668973249024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830074.041011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973249024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830074.041018:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830074.041021:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830074.041024:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973249024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830074.041028:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973249152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830074.041043:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830074.041051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830074/real 1787830074] req@ffff9c1a7895fc00 x1874668973249152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830074.041077:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973249152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830074.041085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830074.041088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830074.041091:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973249152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830074.041096:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973249280:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830074.041112:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830074.041121:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830074/real 1787830074] req@ffff9c1a7895cb40 x1874668973249280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830074.041138:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973249280/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830074.041146:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830074.041148:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830074.041151:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973249280:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830074.041156:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973249408:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830074.041170:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830074.041178:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830074/real 1787830074] req@ffff9c1a7895e1c0 x1874668973249408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830074.041188:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973249408/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830074.041195:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830074.041198:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830074.041201:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973249408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830074.551765:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830074.615586:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830074.807754:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830075.575547:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830075.639863:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830075.833403:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830076.599462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830076.663461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830076.855738:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830077.624445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830077.687523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830077.879560:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830078.648851:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830078.711484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830078.905022:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830079.159518:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830079.159528:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830079.159533:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830079.159542:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6529 00000100:00080000:1.0:1787830079.159552:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830079.159559:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830079.159596:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973249536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830079.159677:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973249536:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787830079.159733:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830079.159765:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830079.159769:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830079.159771:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830079.159775:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6529 00000100:00080000:1.0:1787830079.159780:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830079.159784:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830079.159793:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830079.159806:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973249664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830079.159877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830079/real 1787830079] req@ffff9c1a78a792c0 x1874668973249536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830079.159893:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973249536/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830079.159901:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830079.159905:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830079.159909:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973249536:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830079.159932:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973249664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830079.159955:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830079.159967:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830079/real 1787830079] req@ffff9c1a78a78000 x1874668973249664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830079.159977:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973249664/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830079.159984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830079.159987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830079.159990:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973249664:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830079.160076:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830079.160080:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830079.160082:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830079.160086:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6529 00000100:00080000:1.0:1787830079.160097:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830079.160102:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830079.160107:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830079.160120:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973249792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830079.160135:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830079.160138:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830079.160139:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830079.160143:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6529 00000100:00080000:1.0:1787830079.160146:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830079.160149:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830079.160153:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830079.160161:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973249920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830079.160168:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830079.160171:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830079.160173:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830079.160175:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6529 00000100:00080000:1.0:1787830079.160178:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830079.160182:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830079.160186:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830079.160195:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973250048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830079.160202:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830079.160204:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830079.160206:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830079.160209:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6529 00000100:00080000:1.0:1787830079.160212:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830079.160215:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830079.160219:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830079.160226:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973250176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830079.160232:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830079.160235:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830079.160236:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830079.160239:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6529 00000100:00080000:1.0:1787830079.160242:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830079.160244:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830079.160248:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830079.160254:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973250304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830079.160320:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973249792:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830079.160342:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830079.160381:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830079/real 1787830079] req@ffff9c1a78a79a40 x1874668973249792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830079.160392:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973249792/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830079.160399:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830079.160402:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830079.160405:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973249792:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830079.160410:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973249920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830079.160427:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830079.160436:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830079/real 1787830079] req@ffff9c1a78a7b480 x1874668973249920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830079.160446:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973249920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830079.160453:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830079.160456:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830079.160459:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973249920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830079.160464:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973250048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830079.160480:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830079.160489:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830079/real 1787830079] req@ffff9c1a78a78f00 x1874668973250048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830079.160505:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973250048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830079.160512:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830079.160514:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830079.160517:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973250048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830079.160522:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973250176:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830079.160541:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830079.160550:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830079/real 1787830079] req@ffff9c1a78a7a580 x1874668973250176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830079.160560:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973250176/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830079.160567:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830079.160570:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830079.160572:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973250176:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830079.160577:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973250304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830079.160592:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830079.160601:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830079/real 1787830079] req@ffff9c1a78a7b840 x1874668973250304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830079.160610:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973250304/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830079.160622:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830079.160625:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830079.160628:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973250304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830079.671539:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830079.737090:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830079.927511:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830080.697202:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830080.759479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830080.952006:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830081.719463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830081.783826:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830081.975599:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830082.744063:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830082.807515:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830082.999776:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830083.767499:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830083.831511:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830084.026862:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830084.283593:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830084.283607:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830084.283613:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830084.283625:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6534 00000100:00080000:0.0:1787830084.283638:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830084.283646:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830084.283695:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973250432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830084.283732:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830084.283736:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830084.283738:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830084.283742:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6534 00000100:00080000:0.0:1787830084.283746:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830084.283750:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830084.283763:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830084.283779:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973250560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830084.283789:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830084.283793:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830084.283795:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830084.283798:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6534 00000100:00080000:0.0:1787830084.283802:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830084.283806:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830084.283810:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830084.283819:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973250688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830084.283830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830084.283834:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830084.283835:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830084.283839:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6534 00000100:00080000:0.0:1787830084.283842:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830084.283845:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830084.283849:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830084.283856:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973250816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830084.283864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830084.283868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830084.283869:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830084.283872:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6534 00000100:00100000:1.0:1787830084.283982:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973250432:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830084.284070:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830084.284128:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830084/real 1787830084] req@ffff9c1a78860f00 x1874668973250432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.284147:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973250432/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.284166:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.284170:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830084.284175:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973250432:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830084.284182:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973250560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830084.284202:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830084.284214:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830084/real 1787830084] req@ffff9c1a788621c0 x1874668973250560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.284226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973250560/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.284233:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.284236:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830084.284239:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973250560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830084.284245:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973250688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830084.284260:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830084.284279:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830084/real 1787830084] req@ffff9c1a78863840 x1874668973250688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.284291:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973250688/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.284298:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.284301:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830084.284304:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973250688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830084.284310:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973250816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830084.284329:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830084.284340:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830084/real 1787830084] req@ffff9c1a78862d00 x1874668973250816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.284757:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973250816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.284766:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.284769:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830084.284772:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973250816:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830084.284899:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830084.284904:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830084.284913:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830084.284931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973250944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830084.284965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830084.284968:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830084.284971:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830084.284975:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6534 00000100:00080000:0.0:1787830084.284978:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830084.284982:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830084.284987:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830084.284997:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973251072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830084.285005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830084.285008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830084.285010:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830084.285013:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6534 00000100:00080000:0.0:1787830084.285016:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830084.285020:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830084.285024:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830084.285033:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973251200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830084.289475:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973250944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830084.289578:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830084.289616:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830084/real 1787830084] req@ffff9c1a78860b40 x1874668973250944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.289637:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973250944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.289649:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.289654:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830084.289657:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973250944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830084.289665:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973251072:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830084.289685:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830084.289695:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830084/real 1787830084] req@ffff9c1a78861a40 x1874668973251072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.289707:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973251072/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.289715:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.289718:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830084.289721:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973251072:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830084.289727:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973251200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830084.289741:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830084.289749:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830084/real 1787830084] req@ffff9c1a78862940 x1874668973251200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830084.289760:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973251200/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830084.289768:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830084.289771:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830084.289774:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973251200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830084.791507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830084.921190:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830085.163502:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830085.816085:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830085.943560:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830086.135982:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830086.839467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830086.967482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830087.160423:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830087.863713:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830087.991504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830088.184122:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830088.887472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830089.015495:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830089.207605:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830089.399667:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830089.399679:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830089.399685:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830089.399697:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6539 00000100:00080000:0.0:1787830089.399709:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830089.399716:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830089.399761:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973251328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830089.399934:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973251328:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830089.400007:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830089.400038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830089.400042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830089.400044:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830089.400049:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6539 00000100:00080000:0.0:1787830089.400053:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830089.400057:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830089.400068:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830089.400082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973251456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830089.400092:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830089.400095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830089.400097:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830089.400100:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6539 00000100:00080000:0.0:1787830089.400103:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830089.400106:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830089.400111:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830089.400119:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973251584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830089.400135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830089.400138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830089.400140:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830089.400143:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6539 00000100:00080000:0.0:1787830089.400146:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830089.400149:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830089.400153:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830089.400162:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973251712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830089.400170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830089.400173:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830089.400175:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830089.400178:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6539 00000100:00080000:0.0:1787830089.400181:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830089.400184:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830089.400189:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830089.400202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973251840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830089.400211:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830089.400214:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830089.400216:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830089.400220:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6539 00000100:00080000:0.0:1787830089.400223:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830089.400227:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830089.400388:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830089/real 1787830089] req@ffff9c1a78860780 x1874668973251328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830089.400407:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973251328/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830089.400418:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830089.400422:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830089.400426:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973251328:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830089.400453:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973251456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830089.400476:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830089.400487:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830089/real 1787830089] req@ffff9c1a78860000 x1874668973251456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830089.400499:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973251456/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830089.400507:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830089.400510:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830089.400513:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973251456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830089.400519:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973251584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830089.400536:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830089.400544:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830089/real 1787830089] req@ffff9c1a788630c0 x1874668973251584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830089.400555:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973251584/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787830089.400671:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830089.400686:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973251968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830089.400696:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830089.400699:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830089.400701:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830089.400705:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6539 00000100:00080000:0.0:1787830089.400709:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830089.400712:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830089.400716:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830089.400724:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973252096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830089.400973:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830089.400979:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830089.400983:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973251584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830089.400990:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973251712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830089.401013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830089.401078:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830089/real 1787830089] req@ffff9c1a78862580 x1874668973251712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830089.401090:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973251712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830089.401099:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830089.401102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830089.401105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973251712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830089.401111:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973251840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830089.401130:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830089.401139:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830089/real 1787830089] req@ffff9c1a78861680 x1874668973251840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830089.401150:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973251840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830089.401158:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830089.401161:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830089.401165:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973251840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830089.401184:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973251968:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830089.401204:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830089.401215:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830089/real 1787830089] req@ffff9c1a78861e00 x1874668973251968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830089.401226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973251968/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830089.401246:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830089.401249:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830089.401252:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973251968:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830089.401258:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973252096:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830089.401276:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830089.401288:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830089/real 1787830089] req@ffff9c1a788612c0 x1874668973252096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830089.401299:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973252096/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830089.401306:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830089.401309:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830089.401312:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973252096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830089.911469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830090.039669:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830090.232279:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830090.935459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830091.063855:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830091.258585:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830091.962777:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830092.087761:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830092.279893:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830092.983780:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830093.111498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830093.303628:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830094.007472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830094.135454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830094.329818:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830094.519546:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830094.519556:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830094.519560:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830094.519571:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6544 00000100:00080000:0.0:1787830094.519582:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830094.519589:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830094.519625:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973252224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830094.519666:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830094.519672:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830094.519675:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830094.519681:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6544 00000100:00080000:0.0:1787830094.519688:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830094.519694:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830094.519707:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830094.519724:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973252352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830094.519737:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830094.519741:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830094.519743:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830094.519751:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6544 00000100:00080000:0.0:1787830094.519756:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830094.519761:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830094.519768:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830094.519788:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973252480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830094.519803:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830094.519807:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830094.519810:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830094.519815:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6544 00000100:00080000:0.0:1787830094.519821:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830094.519827:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830094.519832:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830094.519845:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973252608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830094.519855:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830094.519858:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830094.519859:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830094.519863:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6544 00000100:00080000:0.0:1787830094.519866:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830094.519869:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830094.519873:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830094.519882:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973252736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830094.519891:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830094.519894:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830094.519895:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830094.519898:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6544 00000100:00080000:0.0:1787830094.519901:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830094.519905:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830094.519909:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830094.519918:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973252864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830094.519925:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830094.519928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830094.519930:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830094.519933:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6544 00000100:00080000:0.0:1787830094.519936:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830094.519939:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830094.519943:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830094.519953:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973252992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830094.520545:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973252224:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830094.520602:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830094.520638:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830094/real 1787830094] req@ffff9c1a78863c00 x1874668973252224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830094.520654:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973252224/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830094.520665:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830094.520668:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830094.520672:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973252224:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830094.520678:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973252352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830094.520701:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830094.520713:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830094/real 1787830094] req@ffff9c1a788612c0 x1874668973252352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830094.520724:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973252352/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830094.520732:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830094.520735:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830094.520738:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973252352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830094.520743:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973252480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830094.520762:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830094.520772:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830094/real 1787830094] req@ffff9c1a78861e00 x1874668973252480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830094.520782:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973252480/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830094.520788:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830094.520791:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830094.520794:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973252480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830094.520799:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973252608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830094.520817:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830094.520826:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830094/real 1787830094] req@ffff9c1a78861680 x1874668973252608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830094.520837:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973252608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830094.520844:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830094.520847:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830094.520850:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973252608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830094.520855:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973252736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830094.520873:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830094.520882:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830094/real 1787830094] req@ffff9c1a78862580 x1874668973252736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830094.520894:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973252736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830094.520901:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830094.520904:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830094.520907:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973252736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830094.520913:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973252864:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830094.520930:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830094.520940:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830094/real 1787830094] req@ffff9c1a788630c0 x1874668973252864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830094.520952:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973252864/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830094.520959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830094.520962:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830094.520965:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973252864:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830094.520971:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973252992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830094.520986:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830094.520996:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830094/real 1787830094] req@ffff9c1a78860000 x1874668973252992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830094.521005:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973252992/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830094.521012:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830094.521015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830094.521018:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973252992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830095.033070:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830095.159461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830095.351599:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830096.055468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830096.183504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830096.376395:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830097.079834:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830097.207572:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830097.399766:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830098.104021:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830098.232319:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830098.423566:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830099.127478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830099.255485:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830099.448252:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830099.639713:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830099.639727:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830099.639733:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830099.639746:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6549 00000100:00080000:1.0:1787830099.639761:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830099.639771:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830099.639822:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973253120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830099.639858:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830099.639863:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830099.639864:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830099.639869:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6549 00000100:00080000:1.0:1787830099.639873:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830099.639877:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830099.639889:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830099.639901:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973253248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830099.639912:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830099.639915:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830099.639918:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830099.639922:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6549 00000100:00080000:1.0:1787830099.639928:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830099.639933:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830099.639937:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830099.639946:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973253376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830099.639955:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830099.639959:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830099.639961:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830099.639966:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6549 00000100:00080000:1.0:1787830099.639969:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830099.639973:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830099.639976:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830099.639988:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973253504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830099.639998:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830099.640001:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830099.640002:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830099.640007:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6549 00000100:00080000:1.0:1787830099.640011:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830099.640016:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830099.640021:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830099.640035:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973253632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830099.640042:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830099.640045:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830099.640047:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830099.640052:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6549 00000100:00080000:1.0:1787830099.640055:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830099.640059:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830099.640064:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830099.640076:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973253760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830099.640083:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830099.640086:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830099.640088:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830099.640091:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6549 00000100:00080000:1.0:1787830099.640092:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830099.640095:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830099.640100:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830099.640113:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973253888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830099.640229:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973253120:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830099.640341:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973253248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830099.640402:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973253376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830099.640420:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973253504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830099.640450:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973253632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830099.640472:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973253760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830099.640495:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973253888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830100.151409:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830100.279504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830100.471590:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830101.175485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830101.303531:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830101.495622:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830102.199445:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830102.327607:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830102.519644:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787830102.713425:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787830102.713471:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830102.713491:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787830102.714956:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830099/real 1787830102] req@ffff9c1a78a7b840 x1874668973253120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830124 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830102.714984:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973253120/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830124 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830102.714996:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830102.715000:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830102.715004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973253120:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830102.715124:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830102.715153:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830102.715161:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830102.715168:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830102.715179:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830102.715186:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787830102.715245:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830099/real 1787830102] req@ffff9c1a78a7a580 x1874668973253248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830102.715259:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973253248/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830102.715267:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830102.715270:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830102.715274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973253248:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830102.715280:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830099/real 1787830102] req@ffff9c1a78a78f00 x1874668973253376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830102.715296:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973253376/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830102.715304:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830102.715308:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830102.715311:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973253376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830102.715317:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830099/real 1787830102] req@ffff9c1a78a7b480 x1874668973253504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830102.715328:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973253504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830102.715334:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830102.715337:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830102.715339:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973253504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830102.715488:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830099/real 1787830102] req@ffff9c1a78a79a40 x1874668973253632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830102.715503:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973253632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830102.715512:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830102.715516:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830102.715519:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973253632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830102.715526:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830099/real 1787830102] req@ffff9c1a78a78000 x1874668973253760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830102.715538:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973253760/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830102.715546:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830102.715549:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830102.715551:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973253760:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830102.715557:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830099/real 1787830102] req@ffff9c1a78a792c0 x1874668973253888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830102.715567:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973253888/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830102.715575:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830102.715578:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830102.715581:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973253888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830103.223847:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830103.351473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830103.543542:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830104.247483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830104.375619:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830104.567863:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830104.759764:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830104.759773:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830104.759777:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830104.759787:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6555 00000100:00080000:3.0:1787830104.759796:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830104.759802:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830104.759833:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973254016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830104.759983:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973254016:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830104.760045:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830104.760081:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830104.760085:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830104.760088:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830104.760092:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6555 00000100:00080000:3.0:1787830104.760096:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830104.760100:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830104.760109:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830104.760132:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973254144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830104.761292:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830104/real 1787830104] req@ffff9c1a7895da40 x1874668973254016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830104.761314:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973254016/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830104.761323:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830104.761328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830104.761331:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973254016:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787830104.761703:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830104.761707:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830104.761710:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830104.761715:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6555 00000100:00080000:3.0:1787830104.761721:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830104.761727:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830104.761736:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830104.761757:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973254272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830104.761770:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830104.761773:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830104.761774:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830104.761777:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6555 00000100:00080000:3.0:1787830104.761781:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830104.761784:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830104.761789:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830104.761797:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973254400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830104.761804:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830104.761806:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830104.761808:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830104.761811:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6555 00000100:00080000:3.0:1787830104.761814:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830104.761817:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830104.761821:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830104.761828:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973254528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830104.761836:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830104.761839:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830104.761840:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830104.761843:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6555 00000100:00080000:3.0:1787830104.761846:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830104.761849:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830104.761853:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830104.761861:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973254656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830104.761868:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830104.761871:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830104.761873:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830104.761876:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6555 00000100:00080000:3.0:1787830104.761878:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830104.761881:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830104.761884:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830104.761893:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973254784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830104.762041:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973254144:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830104.762093:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830104.762112:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830104/real 1787830104] req@ffff9c1a7895e1c0 x1874668973254144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830104.762134:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973254144/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830104.762148:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830104.762151:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830104.762155:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973254144:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830104.762170:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973254272:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830104.762190:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830104.762201:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830104/real 1787830104] req@ffff9c1a7895cb40 x1874668973254272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830104.762212:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973254272/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830104.762220:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830104.762287:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830104.762291:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973254272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830104.762297:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973254400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830104.762317:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830104.762329:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830104/real 1787830104] req@ffff9c1a7895fc00 x1874668973254400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830104.762340:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973254400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830104.762433:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830104.762437:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830104.762441:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973254400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830104.762447:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973254528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830104.762467:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830104.762478:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830104/real 1787830104] req@ffff9c1a7895f0c0 x1874668973254528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830104.762489:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973254528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830104.762497:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830104.762500:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830104.762503:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973254528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830104.762508:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973254656:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830104.762527:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830104.764008:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830104/real 1787830104] req@ffff9c1a7895f840 x1874668973254656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830104.764031:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973254656/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830104.764041:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830104.764046:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830104.764049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973254656:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830104.764058:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973254784:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830104.764098:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830104.764114:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830104/real 1787830104] req@ffff9c1a7895de00 x1874668973254784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830104.764137:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973254784/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830104.764146:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830104.764149:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830104.764151:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973254784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830105.271946:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830105.399553:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830105.591647:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830106.295821:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830106.423504:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830106.616172:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830107.319487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830107.447467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830107.639543:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830108.343508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830108.471461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830108.664147:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830109.367452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830109.495822:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830109.688061:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830109.880224:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830109.880234:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830109.880239:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830109.880248:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6560 00000100:00080000:1.0:1787830109.880259:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830109.880266:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830109.880303:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973254912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830109.880372:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973254912:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830109.880445:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830109.880594:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830109.880598:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830109.880600:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830109.880604:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6560 00000100:00080000:1.0:1787830109.880609:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830109.880613:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830109.880622:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830109.880636:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973255040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830109.881408:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830109/real 1787830109] req@ffff9c1a78a7b840 x1874668973254912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830109.881427:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973254912/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830109.881438:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830109.881442:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830109.881446:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973254912:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830109.881469:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973255040:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830109.881499:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830109.881513:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830109/real 1787830109] req@ffff9c1a78a7bc00 x1874668973255040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830109.881531:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973255040/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830109.881538:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830109.881541:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830109.881545:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973255040:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830109.881609:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830109.881613:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830109.881615:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830109.881620:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6560 00000100:00080000:1.0:1787830109.881626:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830109.881630:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830109.881636:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830109.881650:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973255168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830109.881662:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830109.881665:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830109.881667:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830109.881670:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6560 00000100:00080000:1.0:1787830109.881674:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830109.881677:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830109.881682:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830109.881691:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973255296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830109.881704:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830109.881707:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830109.881708:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830109.881711:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6560 00000100:00080000:1.0:1787830109.881715:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830109.881718:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830109.881722:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830109.881731:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973255424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830109.881740:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830109.881742:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830109.881744:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830109.881747:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6560 00000100:00080000:1.0:1787830109.881749:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830109.881753:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830109.881758:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830109.881767:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973255552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830109.881775:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830109.881778:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830109.881779:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830109.881783:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6560 00000100:00080000:1.0:1787830109.881786:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830109.881790:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830109.881794:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830109.881803:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973255680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830109.881924:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973255168:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830109.881948:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830109.881960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830109/real 1787830109] req@ffff9c1a78a78780 x1874668973255168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830109.881972:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973255168/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830109.881980:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830109.881984:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830109.881987:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973255168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830109.881993:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973255296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830109.882011:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830109.882020:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830109/real 1787830109] req@ffff9c1a78a7a940 x1874668973255296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830109.882030:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973255296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830109.882043:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830109.882045:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830109.882048:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973255296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830109.882054:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973255424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830109.882071:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830109.882081:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830109/real 1787830109] req@ffff9c1a78a79e00 x1874668973255424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830109.882104:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973255424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830109.882112:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830109.882115:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830109.882118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973255424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830109.882123:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973255552:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830109.882140:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830109.882149:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830109/real 1787830109] req@ffff9c1a78a783c0 x1874668973255552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830109.882159:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973255552/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830109.882166:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830109.882169:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830109.882172:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973255552:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830109.882177:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973255680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830109.882192:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830109.882202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830109/real 1787830109] req@ffff9c1a78a7a1c0 x1874668973255680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830109.882212:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973255680/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830109.882219:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830109.882222:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830109.882289:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973255680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830110.392032:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830110.519976:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830110.711706:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830111.415466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830111.543994:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830111.735605:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830112.439461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830112.568953:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830112.759570:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830113.463468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830113.591481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830113.785189:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830114.487756:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830114.615460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830114.807640:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830114.999590:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830114.999599:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830114.999604:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830114.999614:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6565 00000100:00080000:1.0:1787830114.999685:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830114.999692:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830114.999728:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973255808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830114.999857:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973255808:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787830115.000065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830115.000069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830115.000071:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830115.000076:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6565 00000100:00080000:1.0:1787830115.000080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830115.000085:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830115.000095:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830115.000110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973255936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830115.000121:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830115.000124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830115.000126:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830115.000129:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6565 00000100:00080000:1.0:1787830115.000133:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830115.000136:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830115.000140:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830115.000148:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973256064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830115.000157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830115.000160:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830115.000162:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830115.000165:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6565 00000100:00080000:1.0:1787830115.000168:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830115.000172:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830115.000176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830115.000185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973256192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830115.000192:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830115.000195:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830115.000197:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830115.000199:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6565 00000100:00080000:1.0:1787830115.000202:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830115.000205:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830115.000209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830115.000218:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973256320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830115.000226:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830115.000228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830115.000230:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830115.000233:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6565 00000100:00080000:1.0:1787830115.000236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830115.000240:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830115.000245:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830115.000254:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973256448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830115.000263:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830115.000266:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830115.000267:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830115.000271:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6565 00000100:00080000:1.0:1787830115.000273:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830115.000277:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830115.000281:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830115.000290:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973256576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830115.000419:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830115.000455:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830114/real 1787830115] req@ffff9c1a78a7b0c0 x1874668973255808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830115.000470:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973255808/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830115.000480:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830115.000484:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830115.000487:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973255808:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830115.000509:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973255936:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830115.000529:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830115.000540:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830114/real 1787830115] req@ffff9c1a78a7a1c0 x1874668973255936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830115.000551:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973255936/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830115.000558:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830115.000561:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830115.000564:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973255936:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830115.000569:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973256064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830115.000587:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830115.000598:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830114/real 1787830115] req@ffff9c1a78a783c0 x1874668973256064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830115.000608:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973256064/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830115.000838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830115.000842:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830115.000846:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973256064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830115.000853:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973256192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830115.000875:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830115.000886:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830115/real 1787830115] req@ffff9c1a78a79e00 x1874668973256192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830115.000903:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973256192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830115.000911:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830115.000914:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830115.000918:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973256192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830115.000923:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973256320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830115.000941:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830115.000952:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830115/real 1787830115] req@ffff9c1a78a7a940 x1874668973256320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830115.000963:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973256320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830115.000971:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830115.000974:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830115.000977:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973256320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830115.000983:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973256448:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830115.001002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830115.001012:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830115/real 1787830115] req@ffff9c1a78a78780 x1874668973256448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830115.001024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973256448/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830115.001037:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830115.001040:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830115.001043:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973256448:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830115.001074:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973256576:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830115.001093:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830115.001104:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830115/real 1787830115] req@ffff9c1a78a7bc00 x1874668973256576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830115.001115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973256576/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830115.001123:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830115.001126:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830115.001129:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973256576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830115.511477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830115.639479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830115.831854:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830116.535522:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830116.663460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830116.855753:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830117.559473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830117.687466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830117.879595:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830118.583468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830118.711461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830118.903857:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830119.607818:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830119.735465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830119.928032:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830120.121217:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830120.121226:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830120.121230:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830120.121240:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6570 00000100:00080000:2.0:1787830120.121251:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830120.121259:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830120.121291:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973256704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830120.121320:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830120.121324:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830120.121325:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830120.121329:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6570 00000100:00080000:2.0:1787830120.121332:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830120.121335:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830120.121359:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830120.121369:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973256832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830120.121378:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830120.121381:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830120.121383:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830120.121386:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6570 00000100:00080000:2.0:1787830120.121389:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830120.121392:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830120.121396:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830120.121404:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973256960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830120.121413:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830120.121415:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830120.121416:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830120.121418:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6570 00000100:00080000:2.0:1787830120.121420:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830120.121423:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830120.121426:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830120.121431:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973257088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830120.121437:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830120.121439:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830120.121441:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830120.121444:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6570 00000100:00080000:2.0:1787830120.121446:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830120.121449:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830120.121453:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830120.121462:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973257216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830120.121470:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830120.121473:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830120.121475:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830120.121478:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6570 00000100:00080000:2.0:1787830120.121481:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830120.121485:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830120.121489:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830120.121499:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973257344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830120.121507:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830120.121509:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830120.121511:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830120.121513:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6570 00000100:00080000:2.0:1787830120.121516:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830120.121520:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830120.121524:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830120.121534:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973257472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830120.121610:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973256704:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830120.121667:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830120.121703:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830120/real 1787830120] req@ffff9c1a49d8cb40 x1874668973256704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830120.121717:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973256704/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830120.121726:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830120.121730:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830120.121733:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973256704:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830120.121739:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973256832:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830120.121759:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830120.121771:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830120/real 1787830120] req@ffff9c1a49d8da40 x1874668973256832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830120.121782:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973256832/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830120.121789:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830120.121792:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830120.121795:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973256832:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830120.121800:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973256960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830120.121817:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830120.121824:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830120/real 1787830120] req@ffff9c1a49d8cf00 x1874668973256960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830120.121831:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973256960/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830120.121836:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830120.121838:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830120.121840:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973256960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830120.121844:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973257088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830120.121859:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830120.121869:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830120/real 1787830120] req@ffff9c1a49d8c000 x1874668973257088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830120.121880:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973257088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830120.121888:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830120.121891:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830120.121899:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973257088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830120.121905:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973257216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830120.121923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830120.121934:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830120/real 1787830120] req@ffff9c1a49d8c780 x1874668973257216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830120.121944:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973257216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830120.121951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830120.121954:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830120.121957:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973257216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830120.121962:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973257344:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830120.121978:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830120.121987:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830120/real 1787830120] req@ffff9c1a49d8de00 x1874668973257344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830120.122002:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973257344/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830120.122009:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830120.122012:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830120.122015:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973257344:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830120.122033:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973257472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830120.122051:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830120.122060:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830120/real 1787830120] req@ffff9c1a49d8e1c0 x1874668973257472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830120.122071:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973257472/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830120.122078:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830120.122080:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830120.122083:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973257472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830120.631646:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830120.759835:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830120.951613:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830121.655470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830121.783621:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830121.975861:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830122.679431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830122.807643:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830122.999673:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830123.703456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830123.831454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830124.023960:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830124.728446:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830124.855467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830125.047930:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830125.239471:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830125.239491:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830125.239495:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830125.239509:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6575 00000100:00080000:2.0:1787830125.239520:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830125.239526:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830125.239561:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973257600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830125.239595:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830125.239600:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787830125.239740:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973257600:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830125.239800:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830125.239837:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830125/real 1787830125] req@ffff9c1a49d8fc00 x1874668973257600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830125.239846:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830125.239851:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6575 00000100:00080000:2.0:1787830125.239855:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830125.239859:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830125.239864:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973257600/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787830125.239867:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787830125.239873:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830125.239877:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830125.239880:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973257728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830125.239881:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973257600:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787830125.239890:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830125.239892:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830125.239894:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830125.239897:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6575 00000100:00080000:2.0:1787830125.239900:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830125.239902:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973257728:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787830125.239903:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830125.239907:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830125.239916:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973257856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830125.239921:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787830125.239926:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830125.239929:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830125.239931:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830125.239932:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830125/real 1787830125] req@ffff9c1a49d8ed00 x1874668973257728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830125.239934:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6575 00000100:00080000:2.0:1787830125.239937:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830125.239940:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830125.239943:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973257728/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787830125.239945:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787830125.239951:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787830125.239953:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973257984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830125.239954:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830125.239957:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973257728:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787830125.239961:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830125.239964:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830125.239965:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830125.239967:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973257856:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787830125.239968:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6575 00000100:00080000:2.0:1787830125.239972:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830125.239975:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830125.239979:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787830125.239994:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830125.240000:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973258112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830125.240005:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830125/real 1787830125] req@ffff9c1a49d8f0c0 x1874668973257856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830125.240009:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830125.240012:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830125.240014:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830125.240016:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973257856/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830125.240017:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6575 00000100:00080000:2.0:1787830125.240021:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830125.240024:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830125.240029:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787830125.240029:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830125.240031:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830125.240034:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973257856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830125.240036:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973258240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830125.240040:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973257984:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787830125.240044:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830125.240047:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830125.240049:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830125.240052:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6575 00000100:00080000:2.0:1787830125.240055:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787830125.240055:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787830125.240058:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830125.240063:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830125.240064:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830125/real 1787830125] req@ffff9c1a49d8c3c0 x1874668973257984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830125.240072:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973258368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830125.240074:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973257984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830125.240082:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830125.240084:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830125.240087:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973257984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830125.240099:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973258112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830125.240116:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830125.240126:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830125/real 1787830125] req@ffff9c1a49d8e580 x1874668973258112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830125.240136:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973258112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830125.240144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830125.240147:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830125.240150:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973258112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830125.240155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973258240:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830125.240172:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830125.240183:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830125/real 1787830125] req@ffff9c1a49d8d680 x1874668973258240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830125.240194:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973258240/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830125.240201:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830125.240203:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830125.240206:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973258240:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830125.240211:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973258368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830125.240228:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830125.240305:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830125/real 1787830125] req@ffff9c1a49d8ed00 x1874668973258368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830125.240317:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973258368/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830125.240325:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830125.240328:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830125.240331:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973258368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830125.751657:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830125.879579:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830126.072011:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830126.775467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830126.903475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830127.096014:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830127.799485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830127.927467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830128.119554:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830128.823492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830128.952125:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830129.143742:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830129.847559:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830129.976577:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830130.167872:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830130.359657:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830130.359675:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830130.359680:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830130.359690:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6580 00000100:00080000:3.0:1787830130.359701:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830130.359708:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830130.359745:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973258496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830130.359772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830130.359776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830130.359778:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830130.359782:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6580 00000100:00080000:3.0:1787830130.359786:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830130.359801:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830130.359810:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830130.359817:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973258496:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830130.359824:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973258624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830130.359832:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830130.359834:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830130.359836:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830130.359838:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6580 00000100:00080000:3.0:1787830130.359841:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830130.359843:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830130.359847:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830130.359854:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973258752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830130.359861:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830130.359863:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830130.359865:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830130.359867:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6580 00000100:00080000:3.0:1787830130.359869:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830130.359870:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830130.359873:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787830130.359877:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830130.359882:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973258880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830130.359889:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830130.359891:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830130.359893:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830130.359895:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6580 00000100:00080000:3.0:1787830130.359898:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830130.359900:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830130.359902:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830130.359908:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973259008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830130.359913:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830130.359915:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830130.359919:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830130.359920:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830130/real 1787830130] req@ffff9c1a7895d2c0 x1874668973258496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830130.359923:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6580 00000100:00080000:3.0:1787830130.359927:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830130.359931:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830130.359939:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830130.359960:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973259136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830130.359962:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973258496/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830130.359982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830130.359985:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830130.359987:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830130.359991:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830130.359995:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787830130.359999:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6580 00000100:00100000:1.0:1787830130.360003:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973258496:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787830130.360006:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830130.360023:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830130.360029:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830130.360032:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973258624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830130.360044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973259264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830130.360060:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830130.360075:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830130/real 1787830130] req@ffff9c1a7895f480 x1874668973258624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830130.360087:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973258624/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830130.360094:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830130.360097:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830130.360101:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973258624:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830130.360106:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973258752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830130.360123:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830130.360133:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830130/real 1787830130] req@ffff9c1a7895c000 x1874668973258752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830130.360143:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973258752/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830130.360150:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830130.360153:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830130.360156:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973258752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830130.360162:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973258880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830130.360180:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830130.360189:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830130/real 1787830130] req@ffff9c1a7895e580 x1874668973258880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830130.360212:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973258880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830130.360221:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830130.360223:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830130.360226:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973258880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830130.360232:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973259008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830130.360250:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830130.360260:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830130/real 1787830130] req@ffff9c1a7895cf00 x1874668973259008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830130.360269:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973259008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830130.360277:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830130.360280:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830130.360282:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973259008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830130.360287:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973259136:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830130.360305:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830130.360314:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830130/real 1787830130] req@ffff9c1a7895e940 x1874668973259136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830130.360331:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973259136/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830130.360338:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830130.360341:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830130.360646:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973259136:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830130.360682:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973259264:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830130.360707:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830130.360718:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830130/real 1787830130] req@ffff9c1a7895c780 x1874668973259264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830130.360728:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973259264/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830130.360735:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830130.360738:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830130.360741:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973259264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830130.871772:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830130.999549:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830131.191627:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830131.895516:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830132.023496:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830132.215589:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830132.919945:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830133.047498:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830133.239682:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830133.944813:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830134.072031:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830134.263609:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830134.968100:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830135.095461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830135.287612:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830135.479710:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830135.479721:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830135.479725:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830135.479734:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6585 00000100:00080000:3.0:1787830135.479743:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830135.479750:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830135.479787:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973259392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830135.479818:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830135.479822:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830135.479823:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830135.479827:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6585 00000100:00080000:3.0:1787830135.479831:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830135.479835:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830135.479844:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830135.479856:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973259520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830135.479864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830135.479868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830135.479869:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830135.479872:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6585 00000100:00080000:3.0:1787830135.479875:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830135.479879:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830135.479883:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830135.479890:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973259648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830135.479900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830135.479903:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830135.479904:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830135.479908:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6585 00000100:00080000:3.0:1787830135.479922:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830135.479927:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830135.479931:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830135.479939:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973259776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830135.479947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830135.479950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830135.479952:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830135.479955:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6585 00000100:00080000:3.0:1787830135.479958:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830135.479960:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830135.479964:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830135.479972:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973259904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830135.479979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830135.479982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830135.479984:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830135.479987:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6585 00000100:00080000:3.0:1787830135.479990:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830135.479993:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830135.479997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830135.480005:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973260032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830135.480013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830135.480015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830135.480017:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830135.480020:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6585 00000100:00080000:3.0:1787830135.480024:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830135.480027:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830135.480035:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830135.480043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973260160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830135.481760:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973259392:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830135.481837:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830135.481899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830135/real 1787830135] req@ffff9c1a7895c3c0 x1874668973259392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830135.481928:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973259392/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830135.481941:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830135.481946:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830135.481950:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973259392:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830135.481958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973259520:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830135.481986:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830135.481999:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830135/real 1787830135] req@ffff9c1a7895c780 x1874668973259520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830135.482010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973259520/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830135.482019:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830135.482023:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830135.482026:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973259520:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830135.482032:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973259648:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830135.482051:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830135.482061:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830135/real 1787830135] req@ffff9c1a7895e940 x1874668973259648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830135.482079:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973259648/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830135.482087:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830135.482090:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830135.482093:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973259648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830135.482099:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973259776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830135.482117:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830135.482128:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830135/real 1787830135] req@ffff9c1a7895cf00 x1874668973259776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830135.482139:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973259776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830135.482146:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830135.482159:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830135.482162:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973259776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830135.482167:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973259904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830135.482186:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830135.482197:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830135/real 1787830135] req@ffff9c1a7895e580 x1874668973259904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830135.482208:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973259904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830135.482220:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830135.482223:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830135.482226:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973259904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830135.482232:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973260032:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830135.482251:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830135.482262:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830135/real 1787830135] req@ffff9c1a7895c000 x1874668973260032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830135.482273:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973260032/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830135.482280:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830135.482283:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830135.482286:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973260032:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830135.482293:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973260160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830135.482325:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830135.482336:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830135/real 1787830135] req@ffff9c1a7895f480 x1874668973260160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830135.482390:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973260160/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830135.482399:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830135.482402:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830135.482405:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973260160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830135.992375:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830136.119508:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830136.311633:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830137.015482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830137.143469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830137.335650:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830138.039530:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830138.167458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830138.359531:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830139.063465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830139.191680:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830139.383531:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830140.087520:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830140.215739:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830140.409401:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830140.599939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830140.599953:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830140.599959:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830140.599972:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6590 00000100:00080000:3.0:1787830140.599986:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830140.599995:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830140.600044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973260288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830140.600084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830140.600088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830140.600090:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830140.600096:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6590 00000100:00080000:3.0:1787830140.600100:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830140.600104:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830140.600116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830140.600132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973260416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830140.600142:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830140.600145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830140.600147:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830140.600151:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6590 00000100:00080000:3.0:1787830140.600154:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830140.600158:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830140.600163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830140.600174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973260544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830140.600185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830140.600188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830140.600190:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830140.600194:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6590 00000100:00080000:3.0:1787830140.600197:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830140.600201:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830140.600205:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830140.600215:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973260672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830140.600225:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830140.600228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830140.600230:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830140.600234:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6590 00000100:00080000:3.0:1787830140.600237:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830140.600241:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830140.600246:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830140.600256:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973260800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830140.600265:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830140.600268:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830140.600270:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830140.600274:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6590 00000100:00080000:3.0:1787830140.600277:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830140.600281:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830140.600286:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830140.600296:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973260928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830140.600304:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830140.600307:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830140.600309:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830140.600312:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6590 00000100:00080000:3.0:1787830140.600316:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830140.600319:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830140.600324:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830140.600333:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973261056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830140.600692:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973260288:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830140.600764:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830140.600798:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830140/real 1787830140] req@ffff9c1a7895d2c0 x1874668973260288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830140.600813:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973260288/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830140.600821:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830140.600824:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830140.600827:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973260288:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830140.600832:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973260416:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830140.600846:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830140.600854:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830140/real 1787830140] req@ffff9c1a7895de00 x1874668973260416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830140.600861:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973260416/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830140.600867:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830140.600869:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830140.600871:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973260416:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830140.600891:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973260544:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830140.600915:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830140.600926:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830140/real 1787830140] req@ffff9c1a7895f840 x1874668973260544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830140.600937:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973260544/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830140.600944:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830140.600947:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830140.600950:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973260544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830140.600956:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973260672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830140.600974:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830140.600985:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830140/real 1787830140] req@ffff9c1a7895f0c0 x1874668973260672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830140.600997:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973260672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830140.601004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830140.601006:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830140.601008:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973260672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830140.601013:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973260800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830140.601032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830140.601043:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830140/real 1787830140] req@ffff9c1a7895fc00 x1874668973260800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830140.601054:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973260800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830140.601062:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830140.601065:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830140.601069:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973260800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830140.601075:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973260928:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830140.601095:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830140.601107:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830140/real 1787830140] req@ffff9c1a7895cb40 x1874668973260928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830140.601118:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973260928/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830140.601125:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830140.601128:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830140.601131:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973260928:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830140.601136:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973261056:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830140.601153:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830140.601162:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830140/real 1787830140] req@ffff9c1a7895e1c0 x1874668973261056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830140.601173:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973261056/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830140.601182:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830140.601185:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830140.601188:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973261056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830141.111535:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830141.239475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830141.431578:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830142.136932:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830142.263523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830142.455923:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830143.159469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830143.287439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830143.479882:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830144.183428:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830144.311463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830144.504010:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830145.207563:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830145.336122:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830145.527564:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830145.719564:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830145.719574:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830145.719577:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830145.719586:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6596 00000100:00080000:1.0:1787830145.719596:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830145.719603:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830145.719636:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973261184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830145.719750:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830145.719753:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830145.719755:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830145.719759:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6596 00000100:00080000:1.0:1787830145.719764:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830145.719768:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830145.719777:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830145.719790:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973261312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830145.719800:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830145.719802:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830145.719804:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830145.719807:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6596 00000100:00080000:1.0:1787830145.719811:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830145.719814:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830145.719818:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830145.719827:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973261440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830145.719837:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830145.719850:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830145.719852:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830145.719856:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6596 00000100:00080000:1.0:1787830145.719859:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830145.719863:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830145.719867:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830145.719877:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973261568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830145.719888:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830145.719892:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830145.719893:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830145.719897:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6596 00000100:00080000:1.0:1787830145.719900:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830145.719902:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830145.719907:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830145.719915:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973261696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830145.719923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830145.719926:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830145.719928:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830145.719931:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6596 00000100:00080000:1.0:1787830145.719934:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830145.719938:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830145.719942:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830145.719952:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973261824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830145.719960:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830145.719962:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830145.719964:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830145.719968:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6596 00000100:00080000:1.0:1787830145.719971:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830145.719974:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830145.719978:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830145.719988:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973261952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830145.720059:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973261184:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830145.720115:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830145.720151:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830145/real 1787830145] req@ffff9c1a78a7b840 x1874668973261184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830145.720167:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973261184/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830145.720176:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830145.720180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830145.720184:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973261184:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830145.720190:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973261312:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830145.720211:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830145.720223:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830145/real 1787830145] req@ffff9c1a78a792c0 x1874668973261312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830145.720234:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973261312/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830145.720243:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830145.720245:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830145.720248:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973261312:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830145.720254:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973261440:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830145.720271:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830145.720281:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830145/real 1787830145] req@ffff9c1a78a78000 x1874668973261440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830145.720293:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973261440/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830145.720301:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830145.720304:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830145.720307:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973261440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830145.720312:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973261568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830145.720329:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830145.720340:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830145/real 1787830145] req@ffff9c1a78a79a40 x1874668973261568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830145.720373:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973261568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830145.720381:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830145.720384:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830145.720388:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973261568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830145.720394:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973261696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830145.720413:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830145.720424:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830145/real 1787830145] req@ffff9c1a78a7b480 x1874668973261696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830145.720435:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973261696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830145.720442:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830145.720445:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830145.720448:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973261696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830145.720454:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973261824:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830145.720473:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830145.720484:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830145/real 1787830145] req@ffff9c1a78a78f00 x1874668973261824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830145.720495:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973261824/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830145.720502:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830145.720505:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830145.720508:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973261824:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830145.720513:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973261952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830145.720531:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830145.720541:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830145/real 1787830145] req@ffff9c1a78a7a580 x1874668973261952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830145.720552:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973261952/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830145.720560:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830145.720562:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830145.720565:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973261952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830146.231976:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830146.359627:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830146.552577:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830147.255582:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830147.383486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830147.577060:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830148.279547:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787830148.407609:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830148.599610:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830149.304914:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830149.432464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830149.625109:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830150.327466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830150.455480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830150.647584:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830150.840128:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830150.840138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830150.840142:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830150.840162:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6601 00000100:00080000:2.0:1787830150.840172:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830150.840179:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830150.840214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973262080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830150.840244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830150.840247:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830150.840249:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830150.840253:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6601 00000100:00080000:2.0:1787830150.840256:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830150.840260:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830150.840269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830150.840280:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973262208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830150.840289:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830150.840292:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830150.840294:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830150.840297:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6601 00000100:00080000:2.0:1787830150.840300:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830150.840304:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830150.840309:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830150.840319:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973262336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830150.840328:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830150.840331:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830150.840332:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830150.840336:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6601 00000100:00080000:2.0:1787830150.840339:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830150.840342:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830150.840379:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830150.840390:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973262464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830150.840400:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830150.840403:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830150.840405:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830150.840409:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6601 00000100:00080000:2.0:1787830150.840412:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830150.840420:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830150.840424:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830150.840434:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973262592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830150.840442:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830150.840445:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830150.840447:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830150.840451:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6601 00000100:00080000:2.0:1787830150.840454:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830150.840457:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830150.840462:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830150.840471:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973262720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830150.840479:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830150.840482:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830150.840483:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830150.840486:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6601 00000100:00080000:2.0:1787830150.840489:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830150.840492:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830150.840496:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830150.840505:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973262848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830150.840619:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973262080:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830150.840675:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830150.840709:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830150/real 1787830150] req@ffff9c1a49d8fc00 x1874668973262080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830150.840725:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973262080/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830150.840735:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830150.840738:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830150.840742:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973262080:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830150.840749:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973262208:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830150.840770:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830150.840781:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830150/real 1787830150] req@ffff9c1a49d8e1c0 x1874668973262208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830150.840799:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973262208/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830150.840818:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830150.840822:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830150.840825:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973262208:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830150.840831:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973262336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830150.840849:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830150.840859:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830150/real 1787830150] req@ffff9c1a49d8de00 x1874668973262336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830150.840870:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973262336/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830150.840877:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830150.840880:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830150.840883:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973262336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830150.840889:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973262464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830150.840905:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830150.840915:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830150/real 1787830150] req@ffff9c1a49d8c780 x1874668973262464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830150.840925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973262464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830150.840940:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830150.840942:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830150.840945:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973262464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830150.840951:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973262592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830150.840968:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830150.840978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830150/real 1787830150] req@ffff9c1a49d8c000 x1874668973262592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830150.840988:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973262592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830150.840995:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830150.840998:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830150.841001:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973262592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830150.841006:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973262720:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830150.841024:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830150.841034:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830150/real 1787830150] req@ffff9c1a49d8cf00 x1874668973262720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830150.841045:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973262720/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830150.841052:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830150.841054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830150.841058:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973262720:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830150.841063:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973262848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830150.841078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830150.841087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830150/real 1787830150] req@ffff9c1a49d8da40 x1874668973262848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830150.841097:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973262848/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830150.841104:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830150.841107:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830150.841110:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973262848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830151.351468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830151.479878:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830151.672766:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830152.375474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830152.503728:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830152.695524:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830153.400111:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830153.527614:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830153.719551:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830154.423470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830154.551460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830154.743827:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830155.447469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830155.577250:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830155.767527:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830155.959467:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830155.959475:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830155.959479:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830155.959487:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6606 00000100:00080000:3.0:1787830155.959497:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830155.959503:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830155.959537:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973262976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830155.959558:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830155.959569:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830155.959571:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830155.959576:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6606 00000100:00080000:3.0:1787830155.959580:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830155.959583:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830155.959593:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830155.959603:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973262976:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830155.959605:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973263104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830155.959614:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830155.959616:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830155.959618:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830155.959621:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6606 00000100:00080000:3.0:1787830155.959623:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830155.959627:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830155.959630:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830155.959638:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973263232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830155.959646:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830155.959649:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830155.959650:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830155.959653:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6606 00000100:00080000:3.0:1787830155.959655:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787830155.959659:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830155.959659:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830155.959663:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830155.959670:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973263360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830155.959678:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830155.959681:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830155.959682:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830155.959685:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6606 00000100:00080000:3.0:1787830155.959688:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830155.959691:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830155.959694:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830155.959702:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973263488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830155.959710:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830155.959713:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830155.959714:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830155.959718:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6606 00000100:00080000:3.0:1787830155.959721:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830155.959724:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830155.959728:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830155.959737:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973263616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830155.959745:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830155.959748:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830155.959750:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830155.959753:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6606 00000100:00100000:1.0:1787830155.959754:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830155/real 1787830155] req@ffff9c1a7895da40 x1874668973262976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830155.959756:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830155.959759:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830155.959763:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830155.959780:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973262976/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830155.959781:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973263744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830155.959791:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830155.959795:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830155.959799:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973262976:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830155.959820:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973263104:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830155.959844:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830155.959857:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830155/real 1787830155] req@ffff9c1a7895e1c0 x1874668973263104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830155.959871:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973263104/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830155.959876:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830155.959878:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830155.959880:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973263104:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830155.959892:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973263232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830155.959906:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830155.959913:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830155/real 1787830155] req@ffff9c1a7895cb40 x1874668973263232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830155.959919:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973263232/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830155.959924:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830155.959926:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830155.959928:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973263232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830155.959931:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973263360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830155.959939:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830155.959944:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830155/real 1787830155] req@ffff9c1a7895fc00 x1874668973263360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830155.959950:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973263360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830155.959959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830155.959961:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830155.959962:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973263360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830155.959965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973263488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830155.959973:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830155.959978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830155/real 1787830155] req@ffff9c1a7895f0c0 x1874668973263488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830155.959984:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973263488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830155.959989:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830155.959990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830155.959991:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973263488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830155.959994:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973263616:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830155.960004:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830155.960009:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830155/real 1787830155] req@ffff9c1a7895f840 x1874668973263616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830155.960015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973263616/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830155.960019:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830155.960021:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830155.960023:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973263616:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830155.960026:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973263744:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830155.960034:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830155.960039:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830155/real 1787830155] req@ffff9c1a7895de00 x1874668973263744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830155.960044:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973263744/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830155.960049:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830155.960050:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830155.960052:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973263744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830156.472421:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830156.599487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830156.791809:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830157.495880:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830157.623628:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830157.815796:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830158.519624:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830158.647468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830158.839877:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830159.544416:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830159.672429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830159.863625:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830160.567763:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830160.695784:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830160.887746:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830161.079877:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830161.079888:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830161.079893:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830161.079901:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6611 00000100:00080000:2.0:1787830161.079913:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830161.079920:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830161.079959:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973263872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830161.080030:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973263872:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830161.080083:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787830161.080112:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830161.080116:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830161.080118:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830161.080122:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6611 00000100:00080000:2.0:1787830161.080125:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830161.080129:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830161.080139:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830161.080152:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973264000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830161.080527:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830161/real 1787830161] req@ffff9c1a49d8cb40 x1874668973263872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830161.080546:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973263872/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830161.080556:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830161.080561:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830161.080986:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973263872:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830161.081015:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973264000:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830161.081053:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830161.081069:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830161/real 1787830161] req@ffff9c1a49d8da40 x1874668973264000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830161.081083:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973264000/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830161.081092:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830161.081095:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830161.081098:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973264000:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787830161.081226:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830161.081231:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830161.081234:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830161.081239:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6611 00000100:00080000:2.0:1787830161.081245:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830161.081251:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830161.081259:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830161.081278:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973264128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830161.081322:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973264128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830161.081365:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787830161.081415:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830161.081419:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830161.081422:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830161.081428:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6611 00000100:00080000:2.0:1787830161.081432:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830161.081437:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830161.081442:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830161.081454:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973264256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830161.081525:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830161/real 1787830161] req@ffff9c1a49d8cf00 x1874668973264128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830161.081538:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973264128/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830161.081544:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830161.081547:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830161.081550:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973264128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830161.081637:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973264256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830161.081665:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830161.081677:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830161/real 1787830161] req@ffff9c1a49d8c000 x1874668973264256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830161.081689:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973264256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830161.081697:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830161.081701:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830161.081705:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973264256:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787830161.081769:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830161.081773:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830161.081776:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830161.081848:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6611 00000100:00080000:2.0:1787830161.081852:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830161.081856:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830161.081861:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830161.081874:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973264384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830161.081883:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830161.081886:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830161.081888:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830161.081891:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6611 00000100:00080000:2.0:1787830161.081894:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830161.081898:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830161.081902:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830161.081910:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973264512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830161.081918:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830161.081921:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830161.081923:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830161.081927:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6611 00000100:00080000:2.0:1787830161.081930:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830161.081933:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830161.081938:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830161.081946:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973264640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830161.082110:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973264384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830161.082136:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830161.082150:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830161/real 1787830161] req@ffff9c1a49d8c780 x1874668973264384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830161.082161:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973264384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830161.082169:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830161.082172:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830161.082175:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973264384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830161.082181:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973264512:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830161.082198:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830161.082209:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830161/real 1787830161] req@ffff9c1a49d8de00 x1874668973264512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830161.082219:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973264512/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830161.082227:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830161.082231:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830161.082234:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973264512:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830161.082240:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973264640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830161.082260:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830161.082270:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830161/real 1787830161] req@ffff9c1a49d8e1c0 x1874668973264640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830161.082279:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973264640/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830161.082286:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830161.082289:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830161.082292:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973264640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830161.591568:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830161.719487:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830161.911561:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830162.617974:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830162.744747:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830162.935673:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830163.639465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830163.767650:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830163.960428:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830164.663469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830164.791611:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830164.983607:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830165.687547:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830165.815512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830166.007795:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830166.199519:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830166.199530:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830166.199534:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830166.199544:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6616 00000100:00080000:3.0:1787830166.199558:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830166.199565:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830166.199603:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973264768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830166.199631:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830166.199635:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830166.199638:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830166.199643:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6616 00000100:00080000:3.0:1787830166.199648:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830166.199652:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830166.199664:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830166.199681:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973264768:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830166.199681:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973264896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830166.199713:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830166.199716:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830166.199718:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830166.199723:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6616 00000100:00080000:3.0:1787830166.199727:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830166.199731:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830166.199738:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830166.199751:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973265024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830166.199762:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830166.199765:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830166.199767:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830166.199770:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6616 00000100:00080000:3.0:1787830166.199773:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830166.199777:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830166.199781:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830166.199792:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973265152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830166.199801:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830166.199804:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830166.199806:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830166.199810:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6616 00000100:00080000:3.0:1787830166.199813:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830166.199816:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830166.199822:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830166.199833:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973265280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830166.199841:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973264896:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830166.199845:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830166.199850:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830166.199852:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830166.199856:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6616 00000100:00080000:3.0:1787830166.199860:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830166.199865:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830166.199870:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830166.199878:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973265024:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830166.199884:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973265408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830166.199896:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830166.199899:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830166.199900:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830166.199904:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6616 00000100:00080000:3.0:1787830166.199908:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830166.199913:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830166.199915:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973265152:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787830166.199920:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830166.199936:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973265536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830166.199947:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973265280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830166.199969:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973265408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830166.199987:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973265536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830166.711670:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830166.839469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830167.031945:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830167.736626:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830167.863608:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830168.055656:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830168.759844:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830168.887523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830169.083214:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787830169.273873:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787830169.273954:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787830169.273968:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787830169.275179:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830166/real 1787830169] req@ffff9c1a7895d2c0 x1874668973264768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830191 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830169.275207:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973264768/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830191 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830169.275218:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830169.275223:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830169.275227:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973264768:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787830169.275304:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787830169.275450:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830166/real 1787830169] req@ffff9c1a7895f480 x1874668973264896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830191 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830169.275463:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973264896/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830191 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830169.275471:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830169.275474:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830169.275515:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973264896:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830169.275550:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787830169.275598:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830166/real 1787830169] req@ffff9c1a7895c000 x1874668973265024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830191 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830169.275611:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973265024/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830191 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830169.275617:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830169.275620:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830169.275623:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973265024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830169.275706:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787830169.275816:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830166/real 1787830169] req@ffff9c1a7895e580 x1874668973265152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830191 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830169.275829:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973265152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830191 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830169.275836:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830169.275839:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830169.275842:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973265152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830169.275875:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787830169.275888:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787830169.275958:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:1.0:1787830169.276052:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830166/real 1787830169] req@ffff9c1a7895c780 x1874668973265536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830169.276065:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973265536/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830169.276073:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830169.276077:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830169.276080:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973265536:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830169.276086:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830166/real 1787830169] req@ffff9c1a7895cf00 x1874668973265280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830169.276096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973265280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830169.276103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830169.276106:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830169.276109:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973265280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830169.276115:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830166/real 1787830169] req@ffff9c1a7895e940 x1874668973265408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830169.276125:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973265408/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830169.276133:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830169.276136:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830169.276139:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973265408:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830169.783474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830169.911463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830170.103661:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830170.807750:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830170.935701:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830171.127569:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830171.319530:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830171.319538:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830171.319542:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830171.319551:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6621 00000100:00080000:3.0:1787830171.319560:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830171.319565:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830171.319593:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973265664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830171.319669:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973265664:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787830171.319721:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830171.319725:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830171.319727:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787830171.319730:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830171.319731:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6621 00000100:00080000:3.0:1787830171.319734:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830171.319738:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830171.319748:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830171.319763:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830171/real 1787830171] req@ffff9c1a7895c3c0 x1874668973265664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830171.319763:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973265792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830171.319773:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830171.319777:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787830171.319779:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973265664/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830171.319779:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830171.319782:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6621 00000100:00080000:3.0:1787830171.319785:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830171.319788:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:1.0:1787830171.319789:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830171.319793:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787830171.319793:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830171.319797:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973265664:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830171.319812:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973265920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830171.319818:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973265792:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830171.319823:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830171.319826:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830171.319827:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830171.319831:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6621 00000100:00080000:3.0:1787830171.319833:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787830171.319838:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830171.319838:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830171.319843:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830171.319850:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830171/real 1787830171] req@ffff9c1a7895e940 x1874668973265792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830171.319851:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973266048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830171.319860:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830171.319862:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973265792/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830171.319863:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830171.319865:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830171.319868:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6621 00000100:00080000:1.0:1787830171.319870:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830171.319871:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830171.319873:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787830171.319874:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830171.319876:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973265792:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787830171.319878:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830171.319886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973265920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830171.319886:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973266176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830171.319895:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830171.319898:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830171.319900:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787830171.319902:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830171.319903:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6621 00000100:00080000:3.0:1787830171.319906:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830171.319910:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830171.319911:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830171/real 1787830171] req@ffff9c1a7895cf00 x1874668973265920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830171.319914:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830171.319922:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973265920/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830171.319923:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973266304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830171.319931:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830171.319931:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830171.319934:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787830171.319934:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830171.319936:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830171.319937:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973265920:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830171.319939:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6621 00000100:00100000:1.0:1787830171.319942:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973266048:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830171.319942:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830171.319945:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830171.319949:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787830171.319957:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830171.319957:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973266432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830171.319965:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830171/real 1787830171] req@ffff9c1a7895c780 x1874668973266048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830171.319975:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973266048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830171.319983:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830171.319986:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830171.319989:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973266048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830171.320001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973266176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830171.320017:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830171.320028:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830171/real 1787830171] req@ffff9c1a7895e580 x1874668973266176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830171.320039:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973266176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830171.320046:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830171.320049:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830171.320053:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973266176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:1.0:1787830171.320058:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973266304:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830171.320077:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830171.320087:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830171/real 1787830171] req@ffff9c1a7895c000 x1874668973266304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830171.320098:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973266304/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830171.320105:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830171.320108:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830171.320111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973266304:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830171.320116:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973266432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:1.0:1787830171.320132:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830171.320141:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830171/real 1787830171] req@ffff9c1a7895f480 x1874668973266432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830171.320163:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973266432/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830171.320171:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830171.320175:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:1.0:1787830171.320178:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973266432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830171.831607:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830171.959518:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830172.154730:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830172.855478:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830172.986771:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830173.176109:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830173.879666:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830174.008488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830174.199683:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830174.903459:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830175.031475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830175.223780:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830175.927455:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830176.055831:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830176.247699:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830176.443739:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830176.443748:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830176.443752:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830176.443763:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6626 00000100:00080000:2.0:1787830176.443774:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830176.443780:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830176.443815:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973266560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830176.443846:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830176.443850:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830176.443852:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830176.443856:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6626 00000100:00080000:2.0:1787830176.443860:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830176.443864:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830176.443873:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830176.443884:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973266688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830176.443897:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830176.443900:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830176.443901:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830176.443905:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6626 00000100:00080000:2.0:1787830176.443907:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830176.443910:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830176.443914:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830176.443921:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973266816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830176.443928:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830176.443931:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830176.443932:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830176.443935:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6626 00000100:00080000:2.0:1787830176.443937:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830176.443940:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830176.443943:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830176.443951:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973266944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830176.443958:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830176.443960:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830176.443962:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830176.443965:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6626 00000100:00080000:2.0:1787830176.443967:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830176.443970:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830176.443973:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830176.443981:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973267072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830176.443987:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830176.443990:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830176.443991:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830176.443994:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6626 00000100:00080000:2.0:1787830176.443997:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830176.444001:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830176.444005:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830176.444014:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973267200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830176.444022:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830176.444025:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830176.444027:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830176.444030:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6626 00000100:00080000:2.0:1787830176.444034:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830176.444037:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830176.444041:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830176.444054:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973267328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830176.448244:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973266560:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830176.448306:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830176.448339:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830176/real 1787830176] req@ffff9c1a49d8fc00 x1874668973266560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830176.448387:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973266560/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830176.448399:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830176.448403:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830176.448407:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973266560:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830176.448414:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973266688:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830176.448433:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830176.448443:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830176/real 1787830176] req@ffff9c1a49d8ed00 x1874668973266688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830176.448454:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973266688/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830176.448462:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830176.448465:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830176.448468:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973266688:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830176.448474:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973266816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830176.448499:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830176.448507:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830176/real 1787830176] req@ffff9c1a49d8d680 x1874668973266816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830176.448524:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973266816/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830176.448532:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830176.448535:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830176.448538:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973266816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830176.448543:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973266944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830176.448557:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830176.448565:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830176/real 1787830176] req@ffff9c1a49d8e580 x1874668973266944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830176.448576:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973266944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830176.448583:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830176.448586:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830176.448588:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973266944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830176.448594:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973267072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830176.448611:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830176.448621:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830176/real 1787830176] req@ffff9c1a49d8c3c0 x1874668973267072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830176.448641:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973267072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830176.448654:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830176.448700:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830176.448704:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973267072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830176.448711:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973267200:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830176.448905:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830176.448917:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830176/real 1787830176] req@ffff9c1a49d8f0c0 x1874668973267200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830176.448929:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973267200/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830176.448937:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830176.448940:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830176.448943:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973267200:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830176.448950:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973267328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830176.448970:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830176.448980:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830176/real 1787830176] req@ffff9c1a49d8e1c0 x1874668973267328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830176.448990:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973267328/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830176.448997:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830176.449000:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830176.449004:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973267328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830176.951520:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830177.079436:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830177.272161:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830177.975542:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830178.103459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830178.295594:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830178.999579:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830179.128008:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830179.320404:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830180.023461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830180.151691:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830180.344608:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830181.047468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830181.175466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830181.367568:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830181.563079:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830181.563096:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830181.563100:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830181.563109:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6631 00000100:00080000:2.0:1787830181.563121:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830181.563128:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830181.563168:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973267456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830181.563198:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830181.563202:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830181.563204:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830181.563208:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6631 00000100:00080000:2.0:1787830181.563212:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830181.563215:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830181.563224:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830181.563235:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973267584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830181.563245:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830181.563247:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830181.563249:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830181.563252:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6631 00000100:00080000:2.0:1787830181.563255:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830181.563258:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830181.563263:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830181.563272:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973267712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830181.563282:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830181.563286:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830181.563287:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830181.563290:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6631 00000100:00080000:2.0:1787830181.563293:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830181.563296:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830181.563300:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830181.563309:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973267840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830181.563318:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830181.563322:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830181.563323:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830181.563327:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6631 00000100:00080000:2.0:1787830181.563330:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830181.563333:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830181.563338:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830181.563376:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973267968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830181.563385:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830181.563388:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830181.563390:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830181.563394:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6631 00000100:00080000:2.0:1787830181.563398:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830181.563401:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830181.563406:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830181.563416:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973268096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830181.563424:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830181.563427:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830181.563428:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830181.563431:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6631 00000100:00080000:2.0:1787830181.563434:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830181.563438:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830181.563442:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830181.563452:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973268224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830181.563633:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973267456:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830181.563699:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830181.563733:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830181/real 1787830181] req@ffff9c1a49d8de00 x1874668973267456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830181.563749:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973267456/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830181.563758:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830181.563762:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830181.563766:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973267456:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830181.563773:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973267584:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830181.563794:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830181.563805:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830181/real 1787830181] req@ffff9c1a49d8c780 x1874668973267584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830181.563816:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973267584/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830181.563824:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830181.563827:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830181.563830:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973267584:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830181.563838:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973267712:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830181.563855:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830181.563865:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830181/real 1787830181] req@ffff9c1a49d8c000 x1874668973267712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830181.563877:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973267712/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830181.563884:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830181.563887:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830181.563890:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973267712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830181.563895:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973267840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830181.563910:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830181.563918:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830181/real 1787830181] req@ffff9c1a49d8cf00 x1874668973267840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830181.563928:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973267840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830181.563936:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830181.563938:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830181.563941:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973267840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830181.563946:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973267968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830181.563960:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830181.563969:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830181/real 1787830181] req@ffff9c1a49d8da40 x1874668973267968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830181.563980:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973267968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830181.563987:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830181.563990:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830181.563993:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973267968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830181.563998:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973268096:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830181.564014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830181.564023:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830181/real 1787830181] req@ffff9c1a49d8cb40 x1874668973268096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830181.564034:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973268096/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830181.564041:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830181.564044:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830181.564047:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973268096:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830181.564052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973268224:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830181.564075:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830181.564083:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830181/real 1787830181] req@ffff9c1a49d8e1c0 x1874668973268224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830181.564092:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973268224/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830181.564105:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830181.564108:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830181.564111:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973268224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830182.071603:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830182.199491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830182.391608:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830183.096001:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830183.223482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830183.415783:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830184.119547:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830184.247460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830184.439594:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830185.145730:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830185.271481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830185.464387:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830186.168300:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830186.295466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830186.487562:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830186.680292:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830186.680303:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830186.680307:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830186.680315:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6637 00000100:00080000:1.0:1787830186.680325:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830186.680332:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830186.680410:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973268352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830186.680447:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830186.680451:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830186.680453:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830186.680456:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6637 00000100:00080000:1.0:1787830186.680460:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830186.680463:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830186.680473:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830186.680486:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973268480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830186.680495:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830186.680498:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830186.680500:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830186.680503:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6637 00000100:00080000:1.0:1787830186.680506:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830186.680510:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830186.680514:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830186.680522:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973268608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830186.680531:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830186.680534:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830186.680536:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830186.680539:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6637 00000100:00080000:1.0:1787830186.680541:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830186.680545:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830186.680549:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830186.680569:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973268736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830186.680577:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830186.680580:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830186.680582:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830186.680585:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6637 00000100:00080000:1.0:1787830186.680588:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830186.680591:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830186.680596:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830186.680605:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973268864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830186.680613:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830186.680615:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830186.680616:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830186.680620:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6637 00000100:00080000:1.0:1787830186.680623:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830186.680626:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830186.680630:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830186.680639:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973268992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830186.680646:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830186.680650:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830186.680651:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830186.680654:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6637 00000100:00080000:1.0:1787830186.680658:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830186.680661:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830186.680665:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830186.680675:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973269120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830186.680886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973268352:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787830186.680938:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830186.680975:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830186/real 1787830186] req@ffff9c1a78a7a580 x1874668973268352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830186.680992:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973268352/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830186.681002:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830186.681006:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830186.681010:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973268352:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830186.681016:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973268480:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830186.681033:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830186.681043:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830186/real 1787830186] req@ffff9c1a78a78f00 x1874668973268480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830186.681054:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973268480/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830186.681061:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830186.681064:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830186.681067:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973268480:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830186.681071:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973268608:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830186.681140:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830186.681152:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830186/real 1787830186] req@ffff9c1a78a7b480 x1874668973268608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830186.681164:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973268608/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830186.681171:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830186.681174:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830186.681177:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973268608:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830186.681182:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973268736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830186.681197:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830186.681205:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830186/real 1787830186] req@ffff9c1a78a79a40 x1874668973268736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830186.681214:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973268736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830186.681222:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830186.681225:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830186.681228:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973268736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830186.681232:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973268864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830186.681245:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830186.681252:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830186/real 1787830186] req@ffff9c1a78a78000 x1874668973268864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830186.681261:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973268864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830186.681269:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830186.681327:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830186.681330:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973268864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830186.681336:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973268992:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830186.681425:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830186.681436:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830186/real 1787830186] req@ffff9c1a78a792c0 x1874668973268992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830186.681446:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973268992/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830186.681454:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830186.681456:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830186.681460:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973268992:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830186.681465:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973269120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830186.681480:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830186.681487:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830186/real 1787830186] req@ffff9c1a78a7b840 x1874668973269120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830186.681497:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973269120/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830186.681504:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830186.681506:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830186.681509:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973269120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830187.191576:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830187.319466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830187.511895:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830188.215642:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830188.343621:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830188.535520:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830189.239467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830189.367458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830189.559819:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830190.263427:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830190.391472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830190.583949:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830191.287465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830191.415517:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830191.607572:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830191.799486:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830191.799495:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830191.799500:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830191.799508:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6642 00000100:00080000:3.0:1787830191.799528:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830191.799542:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830191.799577:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973269248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830191.799608:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830191.799611:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830191.799613:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830191.799619:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6642 00000100:00080000:3.0:1787830191.799623:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830191.799627:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830191.799645:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830191.799658:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973269376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830191.799672:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830191.799675:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830191.799676:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830191.799679:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6642 00000100:00080000:3.0:1787830191.799682:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830191.799685:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830191.799689:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830191.799697:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973269504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830191.799704:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830191.799707:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830191.799708:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830191.799711:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6642 00000100:00080000:3.0:1787830191.799722:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830191.799725:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830191.799728:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830191.799746:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973269632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830191.799754:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830191.799758:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830191.799759:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830191.799763:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6642 00000100:00080000:3.0:1787830191.799767:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830191.799771:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830191.799775:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830191.799786:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973269248:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830191.799787:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973269760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830191.799796:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830191.799800:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830191.799805:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830191.799808:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6642 00000100:00080000:3.0:1787830191.799810:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830191.799813:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830191.799817:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830191.799829:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973269888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830191.799836:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830191.799839:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830191.799841:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830191.799844:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6642 00000100:00080000:3.0:1787830191.799848:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830191.799852:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830191.799856:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787830191.799864:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830191.799868:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973270016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830191.799908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830191/real 1787830191] req@ffff9c1a7895d2c0 x1874668973269248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830191.799924:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973269248/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830191.799934:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830191.799939:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830191.799942:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973269248:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830191.799949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973269376:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830191.799971:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830191.799981:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830191/real 1787830191] req@ffff9c1a7895de00 x1874668973269376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830191.799993:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973269376/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830191.800000:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830191.800003:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830191.800006:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973269376:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830191.800012:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973269504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830191.800028:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830191.800037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830191/real 1787830191] req@ffff9c1a7895f840 x1874668973269504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830191.800048:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973269504/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830191.800055:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830191.800058:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830191.800061:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973269504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830191.800067:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973269632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830191.800086:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830191.800096:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830191/real 1787830191] req@ffff9c1a7895f0c0 x1874668973269632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830191.800108:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973269632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830191.800116:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830191.800119:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830191.800122:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973269632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830191.800163:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973269760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830191.800183:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830191.800194:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830191/real 1787830191] req@ffff9c1a7895fc00 x1874668973269760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830191.800204:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973269760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830191.800212:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830191.800215:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830191.800219:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973269760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830191.800224:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973269888:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830191.800243:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830191.800253:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830191/real 1787830191] req@ffff9c1a7895cb40 x1874668973269888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830191.800271:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973269888/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830191.800278:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830191.800281:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830191.800284:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973269888:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830191.800289:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973270016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830191.800307:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830191.800317:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830191/real 1787830191] req@ffff9c1a7895e1c0 x1874668973270016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830191.800328:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973270016/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830191.800336:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830191.800339:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830191.800342:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973270016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830192.312653:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830192.439492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830192.631622:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830193.336748:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830193.463552:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830193.655641:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830194.359489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830194.487465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830194.679842:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830195.383463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830195.511437:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830195.704510:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830196.407596:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830196.535685:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830196.727990:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830196.919527:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830196.919538:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830196.919542:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830196.919552:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6647 00000100:00080000:3.0:1787830196.919562:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830196.919569:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830196.919604:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973270144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830196.919687:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973270144:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787830196.919751:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830196.919751:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830196.919754:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830196.919756:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830196.919760:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6647 00000100:00080000:3.0:1787830196.919764:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830196.919768:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830196.919777:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830196.919785:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830196/real 1787830196] req@ffff9c1a7895da40 x1874668973270144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830196.919790:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973270272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830196.919799:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830196.919801:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973270144/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830196.919802:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830196.919804:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830196.919808:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6647 00000100:00080000:3.0:1787830196.919810:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830196.919811:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830196.919813:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787830196.919814:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787830196.919817:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830196.919818:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973270144:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830196.919825:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973270400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830196.919833:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830196.919836:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830196.919837:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830196.919838:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973270272:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830196.919840:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6647 00000100:00080000:3.0:1787830196.919843:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830196.919846:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830196.919850:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830196.919858:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830196.919858:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973270528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830196.919866:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830196.919868:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830196/real 1787830196] req@ffff9c1a7895e1c0 x1874668973270272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830196.919869:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830196.919871:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830196.919874:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6647 00000100:00080000:3.0:1787830196.919877:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830196.919879:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973270272/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830196.919880:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830196.919885:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830196.919887:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830196.919889:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830196.919892:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973270272:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830196.919898:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973270400:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830196.919914:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830196.919924:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830196/real 1787830196] req@ffff9c1a7895cb40 x1874668973270400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830196.919934:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973270400/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830196.919942:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830196.919945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830196.919948:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973270400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830196.919961:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973270528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830196.919980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830196.919990:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830196/real 1787830196] req@ffff9c1a7895fc00 x1874668973270528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830196.920001:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973270528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830196.920008:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830196.920011:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830196.920014:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973270528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830196.920065:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973270656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830196.920079:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830196.920082:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830196.920084:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830196.920088:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6647 00000100:00080000:3.0:1787830196.920092:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830196.920096:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830196.920101:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830196.920111:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973270656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830196.920112:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973270784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830196.920121:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830196.920123:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830196.920125:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830196.920128:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6647 00000800:00000100:0.0:1787830196.920130:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830196.920131:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830196.920134:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830196.920139:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830196.920140:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830196/real 1787830196] req@ffff9c1a7895f0c0 x1874668973270656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830196.920148:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973270912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830196.920151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973270656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830196.920159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830196.920162:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830196.920165:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973270656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830196.920176:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973270784:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830196.920194:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830196.920204:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830196/real 1787830196] req@ffff9c1a7895f840 x1874668973270784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830196.920215:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973270784/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830196.920222:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830196.920224:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830196.920227:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973270784:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830196.920232:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973270912:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830196.920250:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830196.920259:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830196/real 1787830196] req@ffff9c1a7895de00 x1874668973270912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830196.920270:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973270912/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830196.920278:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830196.920281:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830196.920284:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973270912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830197.431453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830197.559494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830197.751734:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830198.455465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830198.583453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830198.775586:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830199.479456:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830199.607482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830199.800124:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830200.503576:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830200.631532:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830200.823564:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830201.527464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830201.656042:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830201.847950:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830202.039818:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830202.039825:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830202.039830:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830202.039837:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6652 00000100:00080000:3.0:1787830202.039846:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830202.039851:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830202.039881:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973271040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830202.039903:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830202.039906:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830202.039907:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830202.039910:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6652 00000100:00080000:3.0:1787830202.039912:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830202.039914:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830202.039921:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830202.039929:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973271168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830202.039935:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830202.039937:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830202.039939:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830202.039941:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6652 00000100:00080000:3.0:1787830202.039942:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830202.039944:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830202.039947:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830202.039953:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973271040:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830202.039957:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973271296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830202.039969:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830202.039971:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830202.039972:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830202.039974:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6652 00000100:00080000:3.0:1787830202.039976:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830202.039979:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830202.039982:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830202.039989:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973271424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830202.039994:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830202.039996:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830202.039997:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830202.039999:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6652 00000100:00080000:3.0:1787830202.040001:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830202.040003:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830202.040005:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830202.040009:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830202.040013:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973271552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830202.040019:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830202.040021:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830202.040023:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830202.040026:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6652 00000100:00080000:3.0:1787830202.040028:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830202.040031:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830202.040034:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830202.040046:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830202/real 1787830202] req@ffff9c1a7895d2c0 x1874668973271040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830202.040046:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973271680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830202.040075:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830202.040078:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830202.040079:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830202.040082:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973271040/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830202.040084:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6652 00000100:00080000:3.0:1787830202.040089:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830202.040096:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787830202.040097:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830202.040102:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000008:00100000:3.0:1787830202.040103:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830202.040105:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973271040:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830202.040113:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973271168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830202.040119:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973271808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830202.040130:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830202.040140:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830202/real 1787830202] req@ffff9c1a7895f480 x1874668973271168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830202.040148:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973271168/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830202.040153:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830202.040156:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830202.040158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973271168:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830202.040188:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973271296:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830202.040200:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830202.040206:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830202/real 1787830202] req@ffff9c1a7895c000 x1874668973271296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830202.040213:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973271296/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830202.040218:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830202.040220:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830202.040221:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973271296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830202.040225:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973271424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830202.040245:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830202.040251:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830202/real 1787830202] req@ffff9c1a7895e580 x1874668973271424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830202.040258:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973271424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830202.040263:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830202.040265:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830202.040266:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973271424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830202.040270:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973271552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830202.040280:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830202.040285:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830202/real 1787830202] req@ffff9c1a7895c780 x1874668973271552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830202.040291:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973271552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830202.040296:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830202.040297:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830202.040299:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973271552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830202.040302:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973271680:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830202.040311:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830202.040316:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830202/real 1787830202] req@ffff9c1a7895cf00 x1874668973271680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830202.040323:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973271680/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830202.040328:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830202.040329:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830202.040331:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973271680:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830202.040334:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973271808:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830202.040342:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830202.040394:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830202/real 1787830202] req@ffff9c1a7895e940 x1874668973271808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830202.040406:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973271808/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830202.040413:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830202.040416:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830202.040419:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973271808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830202.551482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830202.679494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830202.871556:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830203.575604:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830203.703477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830203.896523:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830204.599483:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830204.727490:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830204.919790:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830205.623479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830205.751473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830205.943810:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830206.647537:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787830206.775480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830206.967576:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830207.159630:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830207.159641:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830207.159654:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830207.159664:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6657 00000100:00080000:2.0:1787830207.159821:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830207.159831:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830207.159869:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973271936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830207.159894:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830207.159898:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830207.159899:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830207.159904:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6657 00000100:00080000:2.0:1787830207.159908:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830207.159911:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830207.159920:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830207.159931:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973272064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830207.159940:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830207.159942:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830207.159944:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830207.159947:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6657 00000100:00080000:2.0:1787830207.159951:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830207.159954:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830207.159958:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830207.159967:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973272192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830207.159976:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830207.159979:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830207.159981:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830207.159984:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6657 00000100:00080000:2.0:1787830207.159987:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830207.159990:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830207.159995:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830207.160004:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973272320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830207.160012:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830207.160015:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830207.160016:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830207.160019:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6657 00000100:00080000:2.0:1787830207.160023:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830207.160026:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830207.160030:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830207.160039:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973272448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830207.160047:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830207.160049:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830207.160051:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830207.160054:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6657 00000100:00080000:2.0:1787830207.160057:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830207.160060:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830207.160065:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830207.160075:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973272576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830207.160090:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830207.160092:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830207.160094:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830207.160097:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6657 00000100:00080000:2.0:1787830207.160099:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830207.160102:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830207.160106:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830207.160115:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973272704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830207.160317:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973271936:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787830207.160465:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830207.160512:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830207/real 1787830207] req@ffff9c1a49d8f0c0 x1874668973271936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830207.160534:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973271936/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830207.160544:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830207.160640:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830207.160760:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973271936:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830207.160769:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973272064:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830207.160796:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830207.160809:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830207/real 1787830207] req@ffff9c1a49d8c3c0 x1874668973272064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830207.160822:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973272064/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830207.160829:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830207.160832:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830207.160836:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973272064:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830207.160841:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973272192:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830207.160859:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830207.160870:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830207/real 1787830207] req@ffff9c1a49d8e580 x1874668973272192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830207.160881:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973272192/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830207.160894:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830207.160898:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830207.160901:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973272192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830207.160906:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973272320:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830207.160925:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830207.160935:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830207/real 1787830207] req@ffff9c1a49d8d680 x1874668973272320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830207.160946:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973272320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830207.160953:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830207.160956:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830207.160958:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973272320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830207.160963:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973272448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830207.160980:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830207.160990:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830207/real 1787830207] req@ffff9c1a49d8ed00 x1874668973272448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830207.161006:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973272448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830207.161012:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830207.161015:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830207.161019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973272448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830207.161024:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973272576:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830207.161043:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830207.161053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830207/real 1787830207] req@ffff9c1a49d8fc00 x1874668973272576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830207.161064:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973272576/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830207.161072:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830207.161075:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830207.161078:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973272576:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830207.161084:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973272704:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830207.161101:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830207.161111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830207/real 1787830207] req@ffff9c1a49d8e1c0 x1874668973272704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830207.161122:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973272704/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830207.161130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830207.161132:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830207.161135:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973272704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830207.671551:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830207.801929:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830207.991662:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830208.695491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830208.823505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830209.017099:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830209.719748:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830209.848407:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830210.039579:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830210.743461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830210.871462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830211.064559:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830211.767471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830211.895526:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830212.088332:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830212.279729:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830212.279739:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830212.279743:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830212.279753:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6662 00000100:00080000:2.0:1787830212.279764:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830212.279771:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830212.279809:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973272832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830212.279844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830212.279848:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830212.279850:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830212.279856:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6662 00000100:00080000:2.0:1787830212.279861:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830212.279865:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830212.279877:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830212.279890:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973272960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830212.279902:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830212.279905:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830212.279907:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830212.279911:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6662 00000100:00080000:2.0:1787830212.279915:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830212.279918:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830212.279923:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830212.279932:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973273088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830212.279942:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830212.279946:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830212.279947:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830212.279950:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6662 00000100:00080000:2.0:1787830212.279953:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830212.279957:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830212.279961:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830212.279970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973273216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830212.279978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830212.279981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830212.279983:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830212.279986:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6662 00000100:00080000:2.0:1787830212.279989:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830212.279993:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830212.279997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830212.280006:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973273344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830212.280014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830212.280017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830212.280018:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830212.280022:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6662 00000100:00080000:2.0:1787830212.280024:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830212.280028:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830212.280032:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830212.280040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973273472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830212.280048:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830212.280051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830212.280052:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830212.280056:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6662 00000100:00080000:2.0:1787830212.280059:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830212.280062:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830212.280066:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830212.280077:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973273600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830212.280326:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973272832:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830212.280496:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830212.280548:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830212/real 1787830212] req@ffff9c1a49d8cb40 x1874668973272832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830212.280569:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973272832/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830212.280579:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830212.280582:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830212.280585:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973272832:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830212.280592:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973272960:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830212.280611:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830212.280620:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830212/real 1787830212] req@ffff9c1a49d8da40 x1874668973272960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830212.280630:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973272960/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830212.280638:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830212.280652:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830212.280656:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973272960:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830212.280662:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973273088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830212.280683:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830212.280694:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830212/real 1787830212] req@ffff9c1a49d8cf00 x1874668973273088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830212.280705:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973273088/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830212.280717:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830212.280720:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830212.280722:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973273088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830212.280739:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973273216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830212.280762:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830212.280775:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830212/real 1787830212] req@ffff9c1a49d8c000 x1874668973273216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830212.280788:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973273216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830212.280798:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830212.280802:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830212.280806:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973273216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830212.280814:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973273344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830212.280846:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830212.280866:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830212/real 1787830212] req@ffff9c1a49d8c780 x1874668973273344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830212.280885:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973273344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830212.280893:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830212.280897:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830212.280901:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973273344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830212.280908:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973273472:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830212.280928:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830212.280939:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830212/real 1787830212] req@ffff9c1a49d8de00 x1874668973273472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830212.280951:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973273472/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830212.280959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830212.280963:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830212.280966:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973273472:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830212.280974:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973273600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830212.280994:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830212.281005:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830212/real 1787830212] req@ffff9c1a49d8e1c0 x1874668973273600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830212.281016:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973273600/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830212.281024:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830212.281027:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830212.281030:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973273600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830212.791720:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830212.919503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830213.111552:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830213.819937:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830213.943453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830214.135576:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830214.839484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830214.967466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830215.160076:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830215.863490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830215.991464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830216.183603:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830216.887507:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830217.015476:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830217.207603:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830217.399486:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830217.399497:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830217.399502:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830217.399512:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6667 00000100:00080000:1.0:1787830217.399522:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830217.399529:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830217.399564:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973273728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830217.399582:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830217.399585:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830217.399586:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830217.399588:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6667 00000100:00080000:1.0:1787830217.399591:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830217.399593:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830217.399600:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830217.399607:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973273856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830217.399613:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830217.399615:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830217.399616:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830217.399618:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6667 00000100:00080000:1.0:1787830217.399619:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830217.399621:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830217.399623:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830217.399628:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973273984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830217.399634:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830217.399636:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830217.399637:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830217.399640:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6667 00000100:00080000:1.0:1787830217.399643:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830217.399646:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830217.399650:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830217.399658:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973274112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830217.399666:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830217.399670:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830217.399671:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830217.399674:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6667 00000100:00080000:1.0:1787830217.399678:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830217.399681:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830217.399685:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830217.399694:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973274240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830217.399703:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830217.399706:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830217.399707:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830217.399712:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6667 00000100:00080000:1.0:1787830217.399715:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830217.399718:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830217.399723:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830217.399732:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973274368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830217.399740:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830217.399743:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830217.399745:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830217.399748:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6667 00000100:00080000:1.0:1787830217.399751:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830217.399754:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830217.399758:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830217.399767:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973274496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830217.400236:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973273728:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830217.400292:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830217.400327:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830217/real 1787830217] req@ffff9c1a78a7bc00 x1874668973273728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.400373:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973273728/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.400384:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.400388:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830217.400391:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973273728:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830217.400398:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973273856:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830217.400421:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830217.400433:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830217/real 1787830217] req@ffff9c1a78a78780 x1874668973273856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.400444:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973273856/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.400452:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.400455:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830217.400458:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973273856:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830217.400465:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973273984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830217.400483:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830217.400493:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830217/real 1787830217] req@ffff9c1a78a7a940 x1874668973273984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.400503:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973273984/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.400509:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.400512:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830217.400515:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973273984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830217.400520:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973274112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830217.400537:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830217.400547:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830217/real 1787830217] req@ffff9c1a78a79e00 x1874668973274112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.400558:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973274112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.400566:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.400569:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830217.400572:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973274112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830217.400657:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973274240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830217.400677:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830217.400687:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830217/real 1787830217] req@ffff9c1a78a783c0 x1874668973274240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.400697:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973274240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.400704:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.400707:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830217.400709:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973274240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830217.400714:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973274368:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830217.400727:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830217.400734:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830217/real 1787830217] req@ffff9c1a78a7a1c0 x1874668973274368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.400743:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973274368/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.400750:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.400753:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830217.400755:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973274368:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830217.400759:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973274496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830217.400770:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830217.400776:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830217/real 1787830217] req@ffff9c1a78a7b0c0 x1874668973274496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830217.400785:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973274496/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830217.400797:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830217.400799:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830217.400801:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973274496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830217.911954:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830218.039456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830218.231761:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830218.936124:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830219.063611:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830219.255951:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830219.959934:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830220.087457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830220.279605:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830220.983439:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830221.111462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830221.305164:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830222.008782:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830222.135450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830222.327541:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830222.519501:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830222.519522:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830222.519526:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830222.519535:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6672 00000100:00080000:1.0:1787830222.519546:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830222.519553:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830222.519594:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973274624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830222.519626:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830222.519630:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830222.519632:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830222.519636:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6672 00000100:00080000:1.0:1787830222.519640:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830222.519656:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830222.519665:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830222.519677:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973274752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830222.519686:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830222.519689:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830222.519691:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830222.519695:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6672 00000100:00080000:1.0:1787830222.519699:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830222.519703:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830222.519707:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830222.519718:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973274880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830222.519727:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830222.519731:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830222.519732:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830222.519735:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6672 00000100:00080000:1.0:1787830222.519739:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830222.519743:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830222.519747:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830222.519757:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973275008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830222.519766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830222.519769:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830222.519770:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830222.519774:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6672 00000100:00080000:1.0:1787830222.519777:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830222.519780:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830222.519785:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830222.519795:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973275136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830222.519804:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830222.519807:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830222.519809:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830222.519812:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6672 00000100:00080000:1.0:1787830222.519816:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830222.519819:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830222.519824:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830222.519834:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973275264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830222.519842:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830222.519844:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830222.519846:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830222.519849:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6672 00000100:00080000:1.0:1787830222.519853:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830222.519856:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830222.519861:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830222.519871:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973275392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830222.520001:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973274624:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830222.520071:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830222.520107:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830222/real 1787830222] req@ffff9c1a78a7b0c0 x1874668973274624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830222.520124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973274624/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830222.520134:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830222.520138:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830222.520142:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973274624:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830222.520149:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973274752:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830222.520172:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830222.520185:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830222/real 1787830222] req@ffff9c1a78a7a1c0 x1874668973274752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830222.520205:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973274752/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830222.520434:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830222.520438:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830222.520442:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973274752:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830222.520448:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973274880:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830222.520473:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830222.520485:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830222/real 1787830222] req@ffff9c1a78a783c0 x1874668973274880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830222.520496:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973274880/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830222.520505:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830222.520508:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830222.520511:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973274880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830222.520517:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973275008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830222.520536:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830222.520548:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830222/real 1787830222] req@ffff9c1a78a79e00 x1874668973275008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830222.520560:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973275008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830222.520574:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830222.520577:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830222.520580:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973275008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830222.520586:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973275136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830222.520606:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830222.520616:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830222/real 1787830222] req@ffff9c1a78a7a940 x1874668973275136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830222.520627:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973275136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830222.520635:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830222.520638:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830222.520641:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973275136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830222.520660:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973275264:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830222.520681:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830222.520693:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830222/real 1787830222] req@ffff9c1a78a78780 x1874668973275264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830222.520705:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973275264/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830222.520713:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830222.520716:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830222.520720:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973275264:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830222.520725:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973275392:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830222.520746:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830222.520758:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830222/real 1787830222] req@ffff9c1a78a7bc00 x1874668973275392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830222.520769:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973275392/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830222.520777:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830222.520779:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830222.520783:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973275392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830223.031472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830223.159471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830223.352119:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830224.055468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830224.183683:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830224.375874:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830225.079437:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830225.207866:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830225.403487:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830226.103602:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830226.231895:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830226.425420:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830227.128196:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830227.257415:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830227.447811:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830227.640285:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830227.640296:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830227.640301:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830227.640312:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6677 00000100:00080000:2.0:1787830227.640324:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830227.640331:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830227.640404:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973275520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830227.640435:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830227.640439:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830227.640441:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830227.640445:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6677 00000100:00080000:2.0:1787830227.640449:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830227.640454:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830227.640465:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830227.640477:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973275648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830227.640488:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830227.640492:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830227.640494:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830227.640498:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6677 00000100:00080000:2.0:1787830227.640501:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830227.640506:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830227.640511:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830227.640520:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973275520:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830227.640525:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973275776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830227.640539:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830227.640542:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830227.640544:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830227.640547:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6677 00000100:00080000:2.0:1787830227.640552:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830227.640555:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830227.640560:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830227.640571:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973275904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830227.640581:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830227.640584:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830227.640586:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787830227.640589:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787830227.640593:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6677 00000100:00080000:2.0:1787830227.640599:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830227.640603:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830227.640608:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830227.640618:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973276032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830227.640627:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830227.640630:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830227.640632:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830227.640637:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830227/real 1787830227] req@ffff9c1a49d8fc00 x1874668973275520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830227.640641:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6677 00000100:00080000:2.0:1787830227.640648:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830227.640655:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830227.640661:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830227.640665:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973275520/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830227.640675:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973276160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830227.640677:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830227.640686:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787830227.640692:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830227.640693:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973275520:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787830227.640700:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830227.640704:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830227.640706:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973275648:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787830227.640721:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6677 00000100:00080000:2.0:1787830227.640730:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787830227.640734:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787830227.640738:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830227.640745:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830227.640752:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830227/real 1787830227] req@ffff9c1a49d8ed00 x1874668973275648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830227.640762:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973276288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830227.640769:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973275648/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830227.640793:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830227.640799:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830227.640804:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973275648:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830227.640834:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973275776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830227.640861:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830227.640877:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830227/real 1787830227] req@ffff9c1a49d8d680 x1874668973275776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830227.640891:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973275776/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830227.640899:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830227.640902:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830227.640905:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973275776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830227.640912:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973275904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830227.641687:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830227.641734:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830227/real 1787830227] req@ffff9c1a49d8e580 x1874668973275904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830227.641749:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973275904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830227.641758:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830227.641762:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830227.641766:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973275904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830227.641773:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973276032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830227.641797:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830227.641808:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830227/real 1787830227] req@ffff9c1a49d8c3c0 x1874668973276032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830227.641819:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973276032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830227.641827:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830227.641830:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830227.641833:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973276032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830227.641840:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973276160:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830227.641860:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830227.641871:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830227/real 1787830227] req@ffff9c1a49d8f0c0 x1874668973276160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830227.641888:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973276160/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830227.641895:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830227.641897:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830227.641900:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973276160:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830227.641906:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973276288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830227.641924:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830227.641934:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830227/real 1787830227] req@ffff9c1a49d8e1c0 x1874668973276288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830227.641945:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973276288/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830227.641957:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830227.641960:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830227.641963:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973276288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830228.151495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830228.279721:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830228.471713:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830229.177413:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830229.303450:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830229.495662:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830230.199464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830230.328996:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830230.520599:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830231.223510:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830231.351566:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830231.543814:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830232.247578:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830232.375469:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830232.568477:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830232.759740:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830232.759749:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830232.759754:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830232.759763:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6683 00000100:00080000:1.0:1787830232.759772:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830232.759778:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830232.759812:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973276416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830232.759869:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973276416:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787830232.760980:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830232.760986:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830232.760988:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830232.760995:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6683 00000100:00080000:1.0:1787830232.761001:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830232.761007:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830232.761017:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830232.761035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973276544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830232.763326:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973276544:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830232.764625:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830232.764634:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830232.764638:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830232.764645:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6683 00000100:00080000:1.0:1787830232.764653:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830232.764661:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830232.764672:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830232.764698:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973276672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830232.764759:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973276672:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830232.764822:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830232.764826:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830232.764828:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830232.764832:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6683 00000100:00080000:1.0:1787830232.764837:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830232.764842:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830232.764847:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830232.764858:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973276800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830232.765341:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973276800:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830232.765745:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830232.765750:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830232.765752:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830232.765758:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6683 00000100:00080000:1.0:1787830232.765763:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830232.765769:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830232.765776:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830232.765793:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973276928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830232.765837:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973276928:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830232.765882:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830232.765886:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830232.765888:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830232.765892:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6683 00000100:00080000:1.0:1787830232.765895:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830232.765899:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830232.765905:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830232.765917:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973277056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830232.765952:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973277056:192.168.203.148@tcp:8:kworker.0 00000100:00080000:1.0:1787830232.766058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830232.766063:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830232.766064:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830232.766068:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6683 00000100:00080000:1.0:1787830232.766072:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830232.766076:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830232.766081:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830232.766092:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973277184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830232.767382:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973277184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830233.271460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830233.399647:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830233.591526:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830234.295531:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830234.423459:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830234.615842:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830235.320102:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830235.447445:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830235.639648:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787830235.831672:0:2400:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:0.0:1787830235.831729:0:2400:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787830235.831746:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787830235.831786:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787830235.831796:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787830235.831802:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787830235.831809:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787830235.831816:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:0.0:1787830235.831822:0:2400:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787830235.832061:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830232/real 1787830235] req@ffff9c1a78a7a580 x1874668973277184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830235.832084:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973277184/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830235.832095:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830235.832100:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830235.832104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973277184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830235.832111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830232/real 1787830235] req@ffff9c1a78a78f00 x1874668973277056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830235.832122:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973277056/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830235.832130:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830235.832133:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830235.832136:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973277056:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830235.832142:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830232/real 1787830235] req@ffff9c1a78a7b480 x1874668973276928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830235.832152:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973276928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830235.832164:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830235.832168:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830235.832171:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973276928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830235.832178:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830232/real 1787830235] req@ffff9c1a78a79a40 x1874668973276800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830235.832188:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973276800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830235.832196:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830235.832200:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830235.832203:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973276800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830235.832209:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830232/real 1787830235] req@ffff9c1a78a78000 x1874668973276672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830235.832232:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973276672/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830235.832238:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830235.832241:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830235.832244:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973276672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830235.832250:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830232/real 1787830235] req@ffff9c1a78a792c0 x1874668973276544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830235.832260:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973276544/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830235.832267:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830235.832270:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830235.832273:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973276544:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830235.832279:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830232/real 1787830235] req@ffff9c1a78a7b840 x1874668973276416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830235.832289:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973276416/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830235.832295:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830235.832298:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830235.832301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973276416:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830236.343671:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830236.471470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830236.664451:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830237.367501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830237.495467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830237.687744:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830237.879843:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830237.879853:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830237.879857:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830237.879868:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6688 00000100:00080000:0.0:1787830237.879880:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830237.879887:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830237.879920:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973277312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830237.879939:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830237.879944:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830237.879947:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830237.879951:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6688 00000100:00080000:0.0:1787830237.879954:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830237.879959:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830237.879965:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973277312:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787830237.879968:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830237.879983:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973277440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830237.879992:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830237.879995:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830237.879997:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830237.880001:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6688 00000100:00080000:0.0:1787830237.880004:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830237.880008:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830237.880013:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830237.880023:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973277568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830237.880033:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830237.880034:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830237.880039:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830237.880041:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830237.880045:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6688 00000100:00080000:0.0:1787830237.880050:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830237.880054:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830237.880058:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830237.880070:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973277696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830237.880077:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830237/real 1787830237] req@ffff9c1a78860780 x1874668973277312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830237.880081:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830237.880089:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830237.880093:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830237.880101:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6688 00000100:00080000:0.0:1787830237.880104:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830237.880110:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830237.880111:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973277312/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787830237.880116:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787830237.880128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787830237.880132:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973277824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830237.880136:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830237.880145:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973277312:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787830237.880149:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830237.880154:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830237.880156:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830237.880160:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6688 00000100:00080000:0.0:1787830237.880164:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830237.880168:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830237.880173:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830237.880176:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973277440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830237.880186:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973277952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830237.880207:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830237.880211:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830237.880213:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830237.880217:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6688 00000800:00000100:2.0:1787830237.880222:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830237.880224:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830237.880231:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830237.880236:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830237.880241:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830237/real 1787830237] req@ffff9c1a78862940 x1874668973277440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830237.880250:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973278080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830237.880260:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973277440/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830237.880270:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830237.880274:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830237.880277:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973277440:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830237.880285:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973277568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830237.880306:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830237.880316:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830237/real 1787830237] req@ffff9c1a78861a40 x1874668973277568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830237.880327:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973277568/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830237.880335:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830237.880338:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830237.880341:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973277568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830237.880369:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973277696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830237.880390:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830237.880402:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830237/real 1787830237] req@ffff9c1a78860b40 x1874668973277696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830237.880419:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973277696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830237.880427:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830237.880430:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830237.880433:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973277696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830237.880438:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973277824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830237.880457:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830237.880467:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830237/real 1787830237] req@ffff9c1a78862d00 x1874668973277824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830237.880479:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973277824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830237.880486:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830237.880489:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830237.880492:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973277824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830237.880524:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973277952:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830237.880545:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830237.880555:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830237/real 1787830237] req@ffff9c1a78863840 x1874668973277952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830237.880566:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973277952/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830237.880579:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830237.880581:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830237.880585:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973277952:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830237.880590:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973278080:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830237.880608:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830237.880618:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830237/real 1787830237] req@ffff9c1a788621c0 x1874668973278080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830237.880629:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973278080/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830237.880637:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830237.880640:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830237.880643:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973278080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830238.391471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830238.519475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830238.712389:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830239.415511:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830239.543477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830239.736239:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830240.439590:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830240.567595:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830240.760609:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830241.463496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830241.591446:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830241.783674:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830242.488674:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830242.615481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830242.808144:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830242.999446:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830242.999457:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830242.999463:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830242.999474:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6693 00000100:00080000:0.0:1787830242.999485:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830242.999492:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830242.999528:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973278208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830242.999558:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830242.999562:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830242.999564:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830242.999568:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6693 00000100:00080000:0.0:1787830242.999572:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830242.999576:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830242.999584:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830242.999598:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973278336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830242.999607:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830242.999611:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830242.999612:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830242.999616:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6693 00000100:00080000:0.0:1787830242.999619:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830242.999623:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830242.999627:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830242.999637:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973278464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830242.999647:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830242.999650:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830242.999651:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830242.999654:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6693 00000100:00080000:0.0:1787830242.999657:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830242.999660:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830242.999664:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830242.999673:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973278592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830242.999680:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830242.999682:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830242.999684:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830242.999687:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6693 00000100:00080000:0.0:1787830242.999689:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830242.999692:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830242.999696:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830242.999703:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973278720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830242.999711:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830242.999724:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830242.999726:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830242.999730:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6693 00000100:00080000:0.0:1787830242.999733:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830242.999736:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830242.999741:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830242.999750:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973278848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830242.999757:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830242.999759:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830242.999761:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830242.999764:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6693 00000100:00080000:0.0:1787830242.999766:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830242.999769:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830242.999772:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830242.999781:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973278976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830242.999839:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973278208:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830242.999896:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830242.999935:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830242/real 1787830242] req@ffff9c1a78860f00 x1874668973278208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830242.999950:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973278208/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830242.999966:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830242.999970:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830242.999974:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973278208:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830242.999980:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973278336:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830243.000000:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830243.000102:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830242/real 1787830243] req@ffff9c1a788621c0 x1874668973278336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830243.000115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973278336/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830243.000123:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830243.000126:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830243.000129:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973278336:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830243.000135:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973278464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830243.000154:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830243.000175:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830242/real 1787830243] req@ffff9c1a78863840 x1874668973278464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830243.000186:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973278464/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830243.000194:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830243.000197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830243.000200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973278464:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830243.000205:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973278592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830243.000231:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830243.000241:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830242/real 1787830243] req@ffff9c1a78862d00 x1874668973278592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830243.000252:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973278592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830243.000268:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830243.000271:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830243.000274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973278592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830243.000279:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973278720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830243.000295:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830243.000304:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830242/real 1787830243] req@ffff9c1a78860b40 x1874668973278720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830243.000314:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973278720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830243.000321:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830243.000324:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830243.000327:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973278720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830243.000332:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973278848:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830243.000367:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830243.000377:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830242/real 1787830243] req@ffff9c1a78861a40 x1874668973278848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830243.000388:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973278848/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830243.000396:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830243.000398:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830243.000401:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973278848:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830243.000406:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973278976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830243.000421:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830243.000439:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830243/real 1787830243] req@ffff9c1a78862940 x1874668973278976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830243.000449:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973278976/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830243.000457:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830243.000460:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830243.000463:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973278976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830243.511505:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830243.639604:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830243.831542:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830244.536223:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830244.663646:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830244.855675:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830245.559472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830245.687463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830245.879578:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830246.583660:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830246.711516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830246.903592:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830247.608372:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830247.736866:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830247.927577:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830248.119594:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830248.119605:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830248.119610:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830248.119619:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6698 00000100:00080000:1.0:1787830248.119631:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830248.119637:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830248.119679:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973279104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830248.119705:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830248.119709:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830248.119711:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830248.119714:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6698 00000100:00080000:1.0:1787830248.119718:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830248.119721:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830248.119732:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830248.119744:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973279232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830248.119753:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830248.119756:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830248.119758:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830248.119761:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6698 00000100:00080000:1.0:1787830248.119764:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830248.119768:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830248.119772:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830248.119782:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973279360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830248.119791:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830248.119794:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830248.119796:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830248.119799:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6698 00000100:00080000:1.0:1787830248.119802:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830248.119805:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830248.119809:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830248.119818:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973279488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830248.119826:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830248.119829:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830248.119831:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830248.119834:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6698 00000100:00080000:1.0:1787830248.119837:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830248.119840:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830248.119844:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830248.119851:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973279616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830248.119859:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830248.119861:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830248.119863:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830248.119866:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6698 00000100:00080000:1.0:1787830248.119868:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830248.119871:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830248.119874:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830248.119881:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973279744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830248.119888:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830248.119891:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830248.119892:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830248.119894:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6698 00000100:00080000:1.0:1787830248.119897:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830248.119899:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830248.119902:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830248.119909:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973279872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830248.120391:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973279104:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830248.120451:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830248.120490:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830248/real 1787830248] req@ffff9c1a78a7b840 x1874668973279104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830248.120516:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973279104/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830248.120527:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830248.120532:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830248.120536:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973279104:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830248.120543:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973279232:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830248.120568:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830248.120583:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830248/real 1787830248] req@ffff9c1a78a792c0 x1874668973279232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830248.120596:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973279232/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830248.120604:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830248.120608:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830248.120611:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973279232:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830248.120617:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973279360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830248.120637:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830248.120648:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830248/real 1787830248] req@ffff9c1a78a78000 x1874668973279360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830248.120660:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973279360/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830248.120673:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830248.120676:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830248.120679:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973279360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830248.120685:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973279488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830248.120706:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830248.120721:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830248/real 1787830248] req@ffff9c1a78a79a40 x1874668973279488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830248.120735:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973279488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830248.120744:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830248.120748:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830248.120753:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973279488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830248.120760:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973279616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830248.120779:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830248.120791:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830248/real 1787830248] req@ffff9c1a78a7b480 x1874668973279616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830248.120803:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973279616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830248.120811:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830248.120814:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830248.120817:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973279616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830248.120823:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973279744:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830248.120844:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830248.120855:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830248/real 1787830248] req@ffff9c1a78a78f00 x1874668973279744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830248.120867:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973279744/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830248.120875:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830248.120878:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830248.120881:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973279744:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830248.120888:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973279872:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830248.120908:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830248.120920:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830248/real 1787830248] req@ffff9c1a78a7a580 x1874668973279872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830248.120933:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973279872/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830248.120942:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830248.120945:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830248.120948:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973279872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830248.631506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830248.761230:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830248.951565:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830249.655488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830249.784341:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830249.975546:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830250.679480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830250.807407:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830251.000426:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830251.703828:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830251.831989:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830252.023885:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830252.727486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830252.855518:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830253.048241:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830253.242111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830253.242123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830253.242127:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830253.242138:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6703 00000100:00080000:3.0:1787830253.242151:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830253.242158:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830253.242195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973280000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830253.242226:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830253.242229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830253.242231:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830253.242235:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6703 00000100:00080000:3.0:1787830253.242239:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830253.242243:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830253.242251:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830253.242264:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973280128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830253.242279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830253.242282:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830253.242283:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830253.242286:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6703 00000100:00080000:3.0:1787830253.242290:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830253.242293:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830253.242297:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830253.242306:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973280256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830253.242315:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830253.242318:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830253.242320:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830253.242323:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6703 00000100:00080000:3.0:1787830253.242326:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830253.242330:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830253.242334:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830253.242342:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973280384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830253.242366:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830253.242369:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830253.242371:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830253.242374:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6703 00000100:00080000:3.0:1787830253.242378:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830253.242382:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830253.242386:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830253.242395:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973280512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830253.242402:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830253.242405:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830253.242406:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830253.242409:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6703 00000100:00080000:3.0:1787830253.242412:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830253.242416:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830253.242420:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830253.242429:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973280640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830253.242437:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830253.242440:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830253.242441:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830253.242444:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6703 00000100:00080000:3.0:1787830253.242448:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830253.242451:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830253.242455:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830253.242463:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973280768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830253.242569:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973280000:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830253.242624:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830253.242660:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830253/real 1787830253] req@ffff9c1a7895c3c0 x1874668973280000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830253.242676:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973280000/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830253.242685:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830253.242690:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830253.242694:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973280000:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830253.242701:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973280128:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830253.242721:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830253.242733:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830253/real 1787830253] req@ffff9c1a7895e940 x1874668973280128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830253.242744:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973280128/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830253.242751:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830253.242754:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830253.242757:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973280128:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830253.242762:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973280256:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830253.242780:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830253.242789:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830253/real 1787830253] req@ffff9c1a7895cf00 x1874668973280256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830253.242800:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973280256/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830253.242812:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830253.242815:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830253.242818:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973280256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830253.242824:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973280384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830253.242841:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830253.242850:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830253/real 1787830253] req@ffff9c1a7895c780 x1874668973280384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830253.242861:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973280384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830253.242869:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830253.242872:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830253.242875:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973280384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830253.242880:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973280512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830253.242895:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830253.242904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830253/real 1787830253] req@ffff9c1a7895e580 x1874668973280512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830253.242914:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973280512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830253.242921:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830253.242923:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830253.242927:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973280512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830253.242932:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973280640:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830253.242948:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830253.243039:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830253/real 1787830253] req@ffff9c1a7895c000 x1874668973280640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830253.243050:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973280640/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830253.243057:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830253.243060:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830253.243063:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973280640:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830253.243068:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973280768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830253.243085:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830253.243247:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830253/real 1787830253] req@ffff9c1a7895f480 x1874668973280768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830253.243260:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973280768/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830253.243267:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830253.243271:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830253.243274:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973280768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830253.751482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830253.879470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830254.073410:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830254.775812:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830254.903457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830255.096047:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830255.799531:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830255.927455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830256.119579:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830256.824142:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830256.951467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830257.143571:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830257.847885:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830257.975706:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830258.168434:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830258.359483:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830258.359495:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830258.359501:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830258.359517:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6708 00000100:00080000:3.0:1787830258.359531:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830258.359540:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830258.359584:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973280896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830258.359668:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973280896:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830258.359731:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830258.359764:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830258.359768:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830258.359770:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830258.359775:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6708 00000100:00080000:3.0:1787830258.359779:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830258.359784:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830258.359793:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830258.359818:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973281024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830258.359931:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830258/real 1787830258] req@ffff9c1a7895d2c0 x1874668973280896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830258.359948:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973280896/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830258.359959:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830258.359964:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830258.359968:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973280896:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830258.359990:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973281024:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830258.360013:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830258.360025:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830258/real 1787830258] req@ffff9c1a7895de00 x1874668973281024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830258.360036:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973281024/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830258.360044:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830258.360047:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830258.360051:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973281024:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830258.360152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830258.360157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830258.360159:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830258.360163:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6708 00000100:00080000:3.0:1787830258.360167:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830258.360171:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830258.360177:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830258.360190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973281152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830258.360235:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973281152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830258.360258:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830258.360280:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830258.360284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830258.360286:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830258.360291:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6708 00000100:00080000:3.0:1787830258.360295:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830258.360299:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830258.360305:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830258.360317:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973281280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830258.360369:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830258/real 1787830258] req@ffff9c1a7895f840 x1874668973281152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830258.360386:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973281152/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830258.360393:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830258.360396:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830258.360399:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973281152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830258.360411:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973281280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830258.360431:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830258.360441:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830258/real 1787830258] req@ffff9c1a7895f0c0 x1874668973281280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830258.360452:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973281280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830258.360458:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830258.360461:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830258.360464:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973281280:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830258.360556:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830258.360561:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830258.360563:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830258.360566:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6708 00000100:00080000:3.0:1787830258.360570:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830258.360574:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830258.360580:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830258.360592:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973281408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830258.360638:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973281408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830258.360660:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830258.360683:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830258.360687:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830258.360689:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830258.360693:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6708 00000100:00080000:3.0:1787830258.360697:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830258.360701:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830258.360706:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830258.360718:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973281536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830258.360755:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830258/real 1787830258] req@ffff9c1a7895fc00 x1874668973281408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830258.360767:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973281408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830258.360780:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830258.360784:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830258.360787:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973281408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830258.360798:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973281536:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830258.360831:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830258.360842:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830258/real 1787830258] req@ffff9c1a7895cb40 x1874668973281536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830258.360852:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973281536/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830258.360860:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830258.360863:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830258.360866:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973281536:192.168.203.148@tcp:8:kworker.0 00000100:00080000:3.0:1787830258.360984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830258.360991:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830258.360994:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830258.360998:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6708 00000100:00080000:3.0:1787830258.361003:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830258.361007:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830258.361012:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830258.361025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973281664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830258.361049:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973281664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830258.361080:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830258.361094:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830258/real 1787830258] req@ffff9c1a7895e1c0 x1874668973281664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830258.361105:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973281664/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830258.361113:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830258.361115:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830258.361119:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973281664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830258.871469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830258.999773:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830259.191536:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830259.895476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830260.023463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830260.215538:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830260.919743:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830261.047465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830261.240216:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830261.943471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830262.072026:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830262.263642:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830262.967468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830263.095470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830263.287761:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830263.479651:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830263.479662:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830263.479666:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830263.479676:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6713 00000100:00080000:1.0:1787830263.479687:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830263.479694:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830263.479902:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973281792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830263.479931:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830263.479935:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830263.479937:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830263.479941:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6713 00000100:00080000:1.0:1787830263.479945:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830263.479949:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830263.479959:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830263.479972:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973281920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830263.479981:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830263.479984:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830263.479986:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830263.479990:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6713 00000100:00080000:1.0:1787830263.479993:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830263.479997:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830263.480001:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830263.480011:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973282048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830263.480033:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830263.480036:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830263.480038:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830263.480041:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6713 00000100:00080000:1.0:1787830263.480045:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830263.480049:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830263.480054:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830263.480064:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973282176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830263.480072:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830263.480075:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830263.480076:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830263.480178:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973281792:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830263.480235:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830263.480271:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830263/real 1787830263] req@ffff9c1a78a7bc00 x1874668973281792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830263.480286:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973281792/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830263.480295:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830263.480299:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830263.480302:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973281792:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830263.480309:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973281920:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830263.480328:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830263.480340:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830263/real 1787830263] req@ffff9c1a78a78780 x1874668973281920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830263.480372:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973281920/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830263.480380:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830263.480383:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830263.480387:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973281920:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830263.480393:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973282048:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830263.480413:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830263.480424:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830263/real 1787830263] req@ffff9c1a78a7a940 x1874668973282048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830263.480434:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973282048/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830263.480441:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830263.480444:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830263.480448:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973282048:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830263.480453:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973282176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830263.480470:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830263.480480:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830263/real 1787830263] req@ffff9c1a78a79e00 x1874668973282176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830263.480490:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973282176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830263.480498:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830263.480501:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830263.480503:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973282176:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830263.480575:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6713 00000100:00080000:1.0:1787830263.480581:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830263.480586:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830263.480591:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830263.480604:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973282304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830263.480627:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830263.480630:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830263.480632:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830263.480635:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6713 00000100:00080000:1.0:1787830263.480638:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830263.480642:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830263.480646:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830263.480657:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973282432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830263.480665:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830263.480668:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830263.480669:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830263.480672:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6713 00000100:00080000:1.0:1787830263.480675:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830263.480679:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830263.480682:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830263.480691:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973282560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830263.481464:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973282304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830263.481497:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830263.481512:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830263/real 1787830263] req@ffff9c1a78a783c0 x1874668973282304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830263.481525:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973282304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830263.481533:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830263.481536:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830263.481540:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973282304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830263.481546:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973282432:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830263.481565:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830263.481575:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830263/real 1787830263] req@ffff9c1a78a7a1c0 x1874668973282432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830263.481586:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973282432/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830263.481594:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830263.481596:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830263.481599:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973282432:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830263.481605:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973282560:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830263.481622:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830263.481632:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830263/real 1787830263] req@ffff9c1a78a7b0c0 x1874668973282560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830263.481642:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973282560/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830263.481650:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830263.481653:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830263.481655:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973282560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830263.991473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830264.119477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830264.312065:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830265.015468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830265.143481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830265.335599:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830266.039464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830266.167471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830266.359942:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830267.064451:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830267.191539:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830267.383646:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830268.087462:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830268.215439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830268.407534:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830268.599722:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830268.599732:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830268.599736:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830268.599746:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6718 00000100:00080000:0.0:1787830268.599757:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830268.599765:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830268.599800:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973282688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830268.599819:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830268.599821:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830268.599823:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830268.599826:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6718 00000100:00080000:0.0:1787830268.599828:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830268.599831:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830268.599839:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830268.599847:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973282816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830268.599853:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830268.599856:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830268.599857:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830268.599859:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6718 00000100:00080000:0.0:1787830268.599861:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830268.599863:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830268.599865:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830268.599870:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973282944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830268.599876:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830268.599877:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830268.599878:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830268.599880:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6718 00000100:00080000:0.0:1787830268.599882:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830268.599884:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830268.599886:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830268.599891:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973283072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830268.599896:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830268.599897:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830268.599898:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830268.599900:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6718 00000100:00080000:0.0:1787830268.599901:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830268.599903:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830268.599906:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830268.599911:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973283200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830268.599915:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830268.599917:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830268.599918:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830268.599919:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6718 00000100:00080000:0.0:1787830268.599921:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830268.599923:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830268.599925:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830268.599931:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973283328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830268.599935:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830268.599937:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830268.599938:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830268.599939:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6718 00000100:00080000:0.0:1787830268.599941:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830268.599942:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830268.599944:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830268.599948:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973283456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830268.600587:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973282688:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830268.600652:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830268.600692:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830268/real 1787830268] req@ffff9c1a78860780 x1874668973282688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830268.600710:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973282688/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830268.600721:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830268.600725:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830268.600729:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973282688:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830268.600737:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973282816:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830268.600760:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830268.600772:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830268/real 1787830268] req@ffff9c1a78860000 x1874668973282816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830268.600784:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973282816/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830268.600797:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830268.600801:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830268.600804:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973282816:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830268.600811:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973282944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830268.600829:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830268.600841:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830268/real 1787830268] req@ffff9c1a788630c0 x1874668973282944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830268.600852:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973282944/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830268.600860:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830268.600863:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830268.600866:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973282944:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830268.600873:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973283072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830268.600893:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830268.600904:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830268/real 1787830268] req@ffff9c1a78862580 x1874668973283072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830268.600916:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973283072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830268.600924:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830268.600928:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830268.600931:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973283072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830268.600937:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973283200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830268.600956:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830268.600968:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830268/real 1787830268] req@ffff9c1a78861680 x1874668973283200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830268.600979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973283200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830268.600997:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830268.601000:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830268.601002:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973283200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830268.601008:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973283328:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830268.601027:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830268.601037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830268/real 1787830268] req@ffff9c1a78861e00 x1874668973283328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830268.601049:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973283328/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830268.601055:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830268.601058:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830268.601061:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973283328:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830268.601066:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973283456:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830268.601083:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830268.601094:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830268/real 1787830268] req@ffff9c1a788612c0 x1874668973283456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830268.601105:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973283456/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830268.601112:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830268.601115:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830268.601118:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973283456:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830269.111479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830269.239507:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830269.431757:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830270.136263:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830270.263506:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830270.456482:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830271.159526:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830271.287509:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830271.481296:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830272.183452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830272.311516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830272.503799:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830273.207484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830273.335924:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830273.528881:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830273.719559:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830273.719569:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830273.719573:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830273.719583:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6724 00000100:00080000:0.0:1787830273.719594:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830273.719602:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830273.719636:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973283584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830273.719662:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830273.719666:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830273.719668:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830273.719672:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6724 00000100:00080000:0.0:1787830273.719675:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830273.719679:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830273.719687:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830273.719702:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973283584:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830273.719703:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973283712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830273.719718:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830273.719721:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830273.719723:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830273.719727:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6724 00000100:00080000:0.0:1787830273.719731:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830273.719735:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830273.719740:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830273.719751:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973283840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830273.719761:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830273.719764:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830273.719766:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787830273.719770:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830273.719774:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6724 00000100:00080000:0.0:1787830273.719778:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830273.719782:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830273.719787:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830273.719798:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973283968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830273.719806:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830273.719808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830273.719810:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830273.719813:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6724 00000100:00080000:0.0:1787830273.719822:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830273.719827:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830273/real 1787830273] req@ffff9c1a78863c00 x1874668973283584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830273.719829:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830273.719837:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830273.719851:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973284096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830273.719857:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973283584/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830273.719864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830273.719872:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830273.719876:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830273.719882:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6724 00000100:00080000:2.0:1787830273.719882:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830273.719887:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830273.719891:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787830273.719895:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830273.719899:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973283584:192.168.203.148@tcp:250:kworker.0 00000008:00100000:0.0:1787830273.719902:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830273.719915:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973284224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830273.719926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830273.719926:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973283712:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830273.719934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830273.719937:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830273.719942:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6724 00000100:00080000:0.0:1787830273.719955:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830273.719960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830273.719964:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787830273.719971:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830273.719978:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973284352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830273.719988:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830273/real 1787830273] req@ffff9c1a788612c0 x1874668973283712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830273.720003:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973283712/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830273.720017:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830273.720021:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830273.720025:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973283712:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830273.720032:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973283840:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830273.720052:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830273.720063:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830273/real 1787830273] req@ffff9c1a78861e00 x1874668973283840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830273.720086:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973283840/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830273.720094:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830273.720097:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830273.720101:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973283840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830273.720107:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973283968:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830273.720127:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830273.720139:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830273/real 1787830273] req@ffff9c1a78861680 x1874668973283968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830273.720151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973283968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830273.720159:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830273.720162:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830273.720165:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973283968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830273.720171:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973284096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830273.720191:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830273.720202:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830273/real 1787830273] req@ffff9c1a78862580 x1874668973284096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830273.720223:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973284096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830273.720232:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830273.720235:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830273.720238:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973284096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830273.720258:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973284224:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830273.720279:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830273.720290:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830273/real 1787830273] req@ffff9c1a788630c0 x1874668973284224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830273.720301:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973284224/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830273.720309:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830273.720311:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830273.720314:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973284224:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830273.720320:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973284352:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830273.720338:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830273.720375:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830273/real 1787830273] req@ffff9c1a78860000 x1874668973284352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830273.720388:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973284352/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830273.720395:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830273.720398:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830273.720401:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973284352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830274.232121:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830274.359429:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830274.551537:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830275.255466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830275.383467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830275.578010:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830276.279466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830276.408086:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830276.600117:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830277.303453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830277.432044:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830277.625996:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830278.327480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830278.456235:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830278.647962:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830278.840713:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830278.840723:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830278.840727:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830278.840739:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6729 00000100:00080000:3.0:1787830278.840749:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830278.840756:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830278.840791:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973284480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830278.840862:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973284480:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830278.840923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830278.840940:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830278.840944:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830278.840946:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830278.840950:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6729 00000100:00080000:3.0:1787830278.840954:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830278.840958:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830278.840976:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830278.840991:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973284608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830278.841052:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830278/real 1787830278] req@ffff9c1a7895da40 x1874668973284480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830278.841068:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830278.841071:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973284480/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830278.841072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830278.841074:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830278.841079:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6729 00000100:00080000:2.0:1787830278.841081:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830278.841082:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830278.841085:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787830278.841086:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830278.841088:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973284480:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787830278.841091:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830278.841101:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973284736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830278.841109:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973284608:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830278.841117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830278.841120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1787830278.841131:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830278.841132:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830278.841136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6729 00000100:00080000:3.0:1787830278.841139:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830278.841142:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830278.841143:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830278/real 1787830278] req@ffff9c1a7895e1c0 x1874668973284608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830278.841147:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830278.841154:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973284608/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830278.841156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973284864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830278.841166:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830278.841168:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830278.841169:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830278.841171:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787830278.841171:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830278.841174:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973284608:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830278.841174:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6729 00000100:00080000:3.0:1787830278.841177:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830278.841180:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830278.841184:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973284736:192.168.203.148@tcp:38:kworker.0 00000008:00100000:3.0:1787830278.841184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830278.841192:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973284992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830278.841200:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830278.841201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830278.841204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830278.841205:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830278.841209:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830278/real 1787830278] req@ffff9c1a7895cb40 x1874668973284736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830278.841209:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6729 00000100:00080000:3.0:1787830278.841212:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830278.841215:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830278.841219:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973284736/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830278.841220:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787830278.841226:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787830278.841228:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973285120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830278.841229:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830278.841232:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973284736:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830278.841236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830278.841238:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973284864:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830278.841240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830278.841241:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830278.841244:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6729 00000100:00080000:3.0:1787830278.841247:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830278.841250:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787830278.841252:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787830278.841254:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830278.841261:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830278/real 1787830278] req@ffff9c1a7895fc00 x1874668973284864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830278.841263:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973285248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830278.841273:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973284864/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830278.841280:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830278.841283:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830278.841286:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973284864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830278.841298:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973284992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830278.841316:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830278.841326:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830278/real 1787830278] req@ffff9c1a7895f0c0 x1874668973284992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830278.841337:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973284992/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830278.841364:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830278.841368:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830278.841371:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973284992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830278.841376:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973285120:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830278.841395:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830278.841405:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830278/real 1787830278] req@ffff9c1a7895f840 x1874668973285120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830278.841490:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973285120/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830278.841495:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830278.841497:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830278.841499:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973285120:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830278.841503:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973285248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830278.841518:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830278.841525:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830278/real 1787830278] req@ffff9c1a7895de00 x1874668973285248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830278.841531:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973285248/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830278.841535:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830278.841537:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830278.841539:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973285248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830279.351448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830279.480324:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830279.671489:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830280.375485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830280.503462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830280.695655:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830281.399466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830281.527630:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830281.720787:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830282.423464:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830282.551462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830282.746297:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830283.447567:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830283.575455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830283.767527:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830283.959456:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830283.959466:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830283.959469:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830283.959477:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6734 00000100:00080000:3.0:1787830283.959488:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830283.959496:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830283.959529:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973285376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830283.959552:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830283.959556:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830283.959557:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830283.959561:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6734 00000100:00080000:3.0:1787830283.959564:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830283.959568:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830283.959576:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830283.959587:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973285504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830283.959596:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830283.959599:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830283.959601:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830283.959604:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6734 00000100:00080000:3.0:1787830283.959607:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830283.959610:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830283.959614:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830283.959623:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973285632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830283.959632:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830283.959635:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830283.959637:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830283.959640:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6734 00000100:00080000:3.0:1787830283.959643:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830283.959646:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830283.959649:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830283.959659:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973285760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830283.959667:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830283.959670:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830283.959671:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830283.959675:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6734 00000100:00080000:3.0:1787830283.959683:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830283.959687:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830283.959691:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830283.959701:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973285888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830283.959709:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830283.959712:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830283.959714:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830283.959716:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6734 00000100:00080000:3.0:1787830283.959720:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830283.959723:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830283.959728:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830283.959795:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973285376:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830283.959798:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973286016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830283.959811:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830283.959814:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830283.959816:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830283.959820:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6734 00000100:00080000:3.0:1787830283.959825:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830283.959830:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830283.959835:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830283.959848:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973286144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830283.959862:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830283.959908:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830283/real 1787830283] req@ffff9c1a7895d2c0 x1874668973285376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830283.959925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973285376/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830283.959932:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830283.959935:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830283.959938:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973285376:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830283.959942:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973285504:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830283.959958:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830283.959967:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830283/real 1787830283] req@ffff9c1a7895f480 x1874668973285504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830283.959976:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973285504/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830283.959990:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830283.959993:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830283.959997:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973285504:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830283.960003:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973285632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830283.960026:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830283.960039:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830283/real 1787830283] req@ffff9c1a7895c000 x1874668973285632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830283.960052:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973285632/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830283.960059:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830283.960062:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830283.960065:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973285632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830283.960071:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973285760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830283.960095:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830283.960111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830283/real 1787830283] req@ffff9c1a7895e580 x1874668973285760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830283.960124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973285760/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830283.960133:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830283.960137:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830283.960140:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973285760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830283.960147:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973285888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830283.960167:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830283.960181:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830283/real 1787830283] req@ffff9c1a7895c780 x1874668973285888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830283.960194:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973285888/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830283.960201:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830283.960204:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830283.960208:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973285888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830283.960242:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973286016:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830283.960262:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830283.960273:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830283/real 1787830283] req@ffff9c1a7895cf00 x1874668973286016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830283.960285:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973286016/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830283.960292:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830283.960293:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830283.960295:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973286016:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830283.960299:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973286144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830283.960311:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830283.960318:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830283/real 1787830283] req@ffff9c1a7895e940 x1874668973286144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830283.960325:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973286144/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830283.960329:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830283.960331:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830283.960332:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973286144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830284.471480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830284.599940:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830284.791562:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830285.495632:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830285.623482:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830285.815543:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830286.519449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830286.647797:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830286.840932:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830287.543508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830287.672067:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830287.865218:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830288.567476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830288.695479:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830288.887640:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830289.079505:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830289.079517:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830289.079521:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830289.079532:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6739 00000100:00080000:3.0:1787830289.079548:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830289.079556:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830289.079600:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973286272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830289.079629:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830289.079633:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830289.079635:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830289.079639:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6739 00000100:00080000:3.0:1787830289.079642:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830289.079647:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830289.079656:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830289.079665:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973286272:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830289.079669:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973286400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830289.079677:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830289.079680:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830289.079682:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830289.079685:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6739 00000100:00080000:3.0:1787830289.079687:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830289.079691:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830289.079695:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830289.079702:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973286528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830289.079710:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830289.079712:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830289.079714:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830289.079717:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6739 00000100:00080000:3.0:1787830289.079720:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787830289.079722:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830289.079723:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830289.079727:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830289.079734:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973286656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830289.079741:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830289.079743:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830289.079745:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830289.079747:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6739 00000100:00080000:3.0:1787830289.079750:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830289.079753:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830289.079757:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830289.079758:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830289/real 1787830289] req@ffff9c1a7895c3c0 x1874668973286272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830289.079765:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973286784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830289.079772:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830289.079773:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973286272/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830289.079775:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830289.079777:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830289.079780:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6739 00000100:00080000:2.0:1787830289.079783:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830289.079783:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830289.079786:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:3.0:1787830289.079786:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830289.079790:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973286272:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787830289.079790:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830289.079798:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973286912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830289.079814:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973286400:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830289.079818:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830289.079821:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830289.079822:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830289.079825:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6739 00000100:00080000:3.0:1787830289.079828:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830289.079831:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787830289.079835:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787830289.079836:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830289.079855:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973287040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830289.079857:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830289/real 1787830289] req@ffff9c1a7895e940 x1874668973286400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830289.079868:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973286400/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830289.079882:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830289.079885:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830289.079888:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973286400:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830289.079894:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973286528:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830289.079923:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830289.079935:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830289/real 1787830289] req@ffff9c1a7895cf00 x1874668973286528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830289.079956:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973286528/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830289.079963:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830289.079967:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830289.079970:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973286528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830289.079976:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973286656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830289.079995:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830289.080005:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830289/real 1787830289] req@ffff9c1a7895c780 x1874668973286656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830289.080015:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973286656/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830289.080022:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830289.080025:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830289.080029:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973286656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830289.080035:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973286784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830289.080052:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830289.080061:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830289/real 1787830289] req@ffff9c1a7895e580 x1874668973286784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830289.080071:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973286784/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830289.080080:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830289.080082:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830289.080085:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973286784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830289.080102:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973286912:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830289.080121:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830289.080131:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830289/real 1787830289] req@ffff9c1a7895c000 x1874668973286912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830289.080141:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973286912/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830289.080148:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830289.080151:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830289.080154:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973286912:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830289.080160:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973287040:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830289.080176:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830289.080186:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830289/real 1787830289] req@ffff9c1a7895f480 x1874668973287040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830289.080197:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973287040/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830289.080205:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830289.080207:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830289.080210:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973287040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830289.592147:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830289.719470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830289.912814:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830290.615503:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830290.743494:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830290.935730:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830291.639602:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830291.767883:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830291.959611:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830292.663813:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830292.791589:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830292.985421:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830293.687496:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830293.815648:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830294.007566:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830294.199447:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830294.199454:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830294.199457:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830294.199467:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6744 00000100:00080000:3.0:1787830294.199477:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830294.199482:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830294.199514:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973287168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830294.199543:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830294.199545:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830294.199547:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830294.199550:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6744 00000100:00080000:3.0:1787830294.199552:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830294.199555:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830294.199562:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973287168:192.168.203.148@tcp:250:kworker.0 00000008:00100000:3.0:1787830294.199563:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830294.199574:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973287296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830294.199581:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830294.199583:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830294.199584:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830294.199586:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6744 00000100:00080000:3.0:1787830294.199589:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830294.199591:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830294.199593:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830294.199603:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973287424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830294.199608:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830294.199610:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830294.199612:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830294.199614:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6744 00000100:00080000:3.0:1787830294.199617:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830294.199619:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830294.199622:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830294.199626:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830294.199632:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973287552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830294.199639:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830294.199641:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830294.199642:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830294.199644:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6744 00000100:00080000:3.0:1787830294.199646:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830294.199647:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830294.199650:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830294.199654:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973287680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830294.199660:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830294.199662:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830294.199663:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830294.199666:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6744 00000100:00100000:2.0:1787830294.199669:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830294/real 1787830294] req@ffff9c1a7895d2c0 x1874668973287168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830294.199670:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830294.199675:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830294.199681:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830294.199691:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973287808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830294.199693:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973287168/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830294.199710:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830294.199712:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830294.199715:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973287168:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787830294.199728:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830294.199731:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830294.199734:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830294.199737:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973287296:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830294.199741:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6744 00000100:00080000:3.0:1787830294.199748:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830294.199752:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830294.199758:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787830294.199759:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830294.199769:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830294/real 1787830294] req@ffff9c1a7895de00 x1874668973287296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830294.199774:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973287936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830294.199784:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973287296/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830294.199795:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830294.199797:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830294.199801:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973287296:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830294.199815:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973287424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830294.199828:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830294.199835:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830294/real 1787830294] req@ffff9c1a7895f840 x1874668973287424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830294.199842:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973287424/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830294.199847:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830294.199849:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830294.199850:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973287424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830294.199853:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973287552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830294.199863:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830294.199868:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830294/real 1787830294] req@ffff9c1a7895f0c0 x1874668973287552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830294.199874:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973287552/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830294.199879:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830294.199880:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830294.199882:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973287552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830294.199886:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973287680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830294.199894:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830294.199899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830294/real 1787830294] req@ffff9c1a7895fc00 x1874668973287680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830294.199905:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973287680/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830294.199909:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830294.199911:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830294.199913:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973287680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830294.199916:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973287808:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830294.199924:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830294.199929:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830294/real 1787830294] req@ffff9c1a7895cb40 x1874668973287808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830294.199936:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973287808/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830294.199940:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830294.199941:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830294.199943:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973287808:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830294.199955:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973287936:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830294.199963:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830294.199968:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830294/real 1787830294] req@ffff9c1a7895e1c0 x1874668973287936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830294.199975:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973287936/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830294.199979:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830294.199981:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830294.199982:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973287936:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830294.711484:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830294.839558:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830295.031614:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830295.735901:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830295.863540:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830296.055566:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830296.759467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830296.888613:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830297.079641:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830297.783443:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830297.911465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830298.103632:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830298.807526:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830298.936073:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830299.128498:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830299.319496:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830299.319507:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830299.319512:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830299.319522:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6749 00000100:00080000:0.0:1787830299.319535:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830299.319543:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830299.319580:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973288064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830299.319608:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830299.319612:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830299.319615:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830299.319619:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6749 00000100:00080000:0.0:1787830299.319622:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830299.319627:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830299.319637:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830299.319649:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973288192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830299.319658:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830299.319661:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830299.319663:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830299.319667:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6749 00000100:00080000:0.0:1787830299.319670:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830299.319673:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830299.319679:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830299.319689:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973288320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830299.319698:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973288064:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787830299.319703:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830299.319708:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830299.319710:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830299.319714:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6749 00000100:00080000:0.0:1787830299.319719:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830299.319723:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830299.319728:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830299.319740:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973288448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830299.319749:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830299.319752:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830299.319754:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830299.319758:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6749 00000100:00080000:0.0:1787830299.319762:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830299.319766:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830299.319780:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830299.319793:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973288576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830299.319799:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973288192:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830299.319807:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830299.319811:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830299.319813:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830299.319819:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6749 00000100:00080000:0.0:1787830299.319823:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830299.319828:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830299.319833:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973288320:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787830299.319834:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830299.319850:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973288704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830299.319853:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973288448:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830299.319869:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830299.319871:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973288576:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830299.319876:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830299.319880:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830299.319885:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6749 00000100:00080000:0.0:1787830299.319889:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830299.319890:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973288704:192.168.203.148@tcp:8:kworker.0 00000100:00080000:0.0:1787830299.319897:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830299.319903:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830299.319916:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973288832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830299.320122:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973288832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830299.831425:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830299.960234:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830300.151551:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830300.855473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830300.983428:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830301.175598:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830301.879563:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830302.007721:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830302.199621:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787830302.391629:0:2398:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787830302.391675:0:2398:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830302.391693:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830302.391734:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830302.391744:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830302.391763:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830302.391771:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830302.391780:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830302.391789:0:2398:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:2.0:1787830302.391801:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830299/real 1787830302] req@ffff9c1a78863840 x1874668973288704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830302.391826:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973288704/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830302.391838:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830302.391842:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830302.391845:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973288704:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830302.391851:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830299/real 1787830302] req@ffff9c1a78861a40 x1874668973288320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830302.391863:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973288320/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830302.391871:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830302.391874:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830302.391878:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973288320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830302.391883:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830299/real 1787830302] req@ffff9c1a78860b40 x1874668973288448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830302.391893:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973288448/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830302.391900:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830302.391902:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830302.391905:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973288448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830302.391911:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830299/real 1787830302] req@ffff9c1a78862d00 x1874668973288576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830302.391921:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973288576/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830302.391928:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830302.391932:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830302.391935:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973288576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830302.391940:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830299/real 1787830302] req@ffff9c1a78860780 x1874668973288064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830302.391950:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973288064/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830302.391956:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830302.391958:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830302.391961:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973288064:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830302.391967:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830299/real 1787830302] req@ffff9c1a78862940 x1874668973288192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830302.391976:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973288192/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830302.391982:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830302.391985:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830302.391988:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973288192:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830302.392016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830299/real 1787830302] req@ffff9c1a788621c0 x1874668973288832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830302.392026:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973288832/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830302.392033:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830302.392036:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830302.392039:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973288832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830302.903449:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830303.031464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830303.225777:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830303.927461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830304.055462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830304.247816:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830304.439481:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830304.439492:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830304.439497:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830304.439508:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6754 00000100:00080000:3.0:1787830304.439518:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830304.439524:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830304.439560:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973288960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830304.439586:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830304.439589:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830304.439591:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830304.439595:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6754 00000100:00080000:3.0:1787830304.439598:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830304.439602:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830304.439611:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830304.439625:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973289088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830304.439634:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830304.439637:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830304.439639:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830304.439642:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6754 00000100:00080000:3.0:1787830304.439645:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830304.439648:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830304.439653:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830304.439663:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973289216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830304.439678:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830304.439681:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830304.439683:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830304.439686:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6754 00000100:00080000:3.0:1787830304.439690:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830304.439694:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830304.439698:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830304.439709:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973289344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830304.439718:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830304.439722:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973288960:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787830304.439723:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830304.439727:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830304.439746:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6754 00000100:00080000:3.0:1787830304.439750:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830304.439754:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830304.439759:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830304.439771:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973289472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830304.439780:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830304.439783:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830304.439785:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830304.439788:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6754 00000100:00080000:3.0:1787830304.439794:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830304.439797:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830304.439803:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787830304.439804:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830304.439817:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973289600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830304.439827:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830304.439830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830304.439832:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830304.439836:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6754 00000100:00080000:3.0:1787830304.439840:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830304.439843:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830304/real 1787830304] req@ffff9c1a7895da40 x1874668973288960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830304.439848:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830304.439856:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830304.439867:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973288960/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830304.439872:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973289728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830304.439892:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830304.439898:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830304.439903:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973288960:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830304.439912:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973289088:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830304.439934:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830304.439945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830304/real 1787830304] req@ffff9c1a7895e1c0 x1874668973289088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830304.439955:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973289088/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830304.439962:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830304.439965:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830304.439967:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973289088:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830304.439972:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973289216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830304.439990:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830304.440000:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830304/real 1787830304] req@ffff9c1a7895cb40 x1874668973289216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830304.440010:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973289216/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830304.440016:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830304.440019:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830304.440022:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973289216:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830304.440048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973289344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830304.440066:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830304.440077:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830304/real 1787830304] req@ffff9c1a7895fc00 x1874668973289344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830304.440088:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973289344/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830304.440095:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830304.440098:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830304.440101:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973289344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830304.440107:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973289472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830304.440125:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830304.440135:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830304/real 1787830304] req@ffff9c1a7895f0c0 x1874668973289472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830304.440151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973289472/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830304.440158:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830304.440160:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830304.440163:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973289472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830304.440168:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973289600:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830304.440192:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830304.440207:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830304/real 1787830304] req@ffff9c1a7895f840 x1874668973289600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830304.440219:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973289600/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830304.440231:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830304.440234:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830304.440237:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973289600:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830304.440242:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973289728:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830304.440259:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830304.440272:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830304/real 1787830304] req@ffff9c1a7895de00 x1874668973289728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830304.440282:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973289728/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830304.440295:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830304.440298:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830304.440301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973289728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830304.951490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830305.079813:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830305.271643:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830305.975560:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830306.103512:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830306.296216:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830306.999548:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830307.127484:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830307.321799:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830308.023472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830308.151438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830308.343798:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830309.047680:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830309.175505:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830309.367650:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830309.559751:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830309.559761:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830309.559764:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830309.559775:0:60751:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6759 00000100:00080000:0.0:1787830309.559786:0:60751:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830309.559792:0:60751:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830309.559832:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973289856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830309.559860:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830309.559864:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830309.559866:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830309.559870:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6759 00000100:00080000:0.0:1787830309.559874:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830309.559877:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830309.559886:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830309.559899:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973289984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830309.559909:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830309.559911:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830309.559913:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830309.559916:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6759 00000100:00080000:0.0:1787830309.559919:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830309.559923:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830309.559927:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830309.559936:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973290112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830309.559944:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830309.559946:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830309.559948:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830309.559951:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6759 00000100:00080000:0.0:1787830309.559954:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830309.559958:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830309.559962:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830309.559971:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973290240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830309.559980:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830309.559983:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830309.559984:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830309.559987:0:60751:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6759 00000100:00080000:0.0:1787830309.559989:0:60751:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830309.559993:0:60751:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830309.559997:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830309.560004:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973290368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830309.560011:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830309.560014:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830309.560016:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830309.560018:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6759 00000100:00080000:0.0:1787830309.560021:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830309.560024:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830309.560029:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830309.560038:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973290496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830309.560045:0:60751:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830309.560048:0:60751:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830309.560050:0:60751:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830309.560053:0:60751:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6759 00000100:00080000:0.0:1787830309.560056:0:60751:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830309.560059:0:60751:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830309.560127:0:60751:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830309.560137:0:60751:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973290624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830309.560285:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973289856:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830309.560342:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830309.560397:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830309/real 1787830309] req@ffff9c1a78860f00 x1874668973289856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830309.560413:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973289856/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830309.560423:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830309.560427:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830309.560430:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973289856:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830309.560437:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973289984:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830309.560457:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830309.560468:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830309/real 1787830309] req@ffff9c1a788621c0 x1874668973289984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830309.560479:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973289984/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830309.560487:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830309.560490:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830309.560494:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973289984:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830309.560500:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973290112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830309.560527:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830309.560539:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830309/real 1787830309] req@ffff9c1a78862940 x1874668973290112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830309.560550:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973290112/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830309.560558:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830309.560560:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830309.560564:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973290112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830309.560570:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973290240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830309.560586:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830309.560595:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830309/real 1787830309] req@ffff9c1a78860780 x1874668973290240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830309.560606:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973290240/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830309.560614:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830309.560617:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830309.560620:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973290240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830309.560626:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973290368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830309.560640:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830309.560649:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830309/real 1787830309] req@ffff9c1a78862d00 x1874668973290368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830309.560660:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973290368/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830309.560668:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830309.560671:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830309.560673:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973290368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830309.560678:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973290496:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830309.560694:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830309.560716:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830309/real 1787830309] req@ffff9c1a78860b40 x1874668973290496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830309.560728:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973290496/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830309.560736:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830309.560747:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830309.560751:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973290496:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830309.560756:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973290624:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830309.560774:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830309.560784:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830309/real 1787830309] req@ffff9c1a78861a40 x1874668973290624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830309.560795:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973290624/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830309.560809:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830309.560813:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830309.560816:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973290624:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830310.071800:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830310.201971:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830310.392038:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830311.095468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830311.223720:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830311.415550:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830312.119912:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830312.247467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830312.439757:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830313.143550:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830313.271443:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830313.463871:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830314.167470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830314.296381:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830314.487690:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830314.679759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830314.679769:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830314.679773:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830314.679782:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6765 00000100:00080000:0.0:1787830314.679793:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830314.679800:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830314.679850:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973290752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830314.679880:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830314.679884:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830314.679886:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830314.679890:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6765 00000100:00080000:0.0:1787830314.679893:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830314.679897:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830314.679906:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830314.679919:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973290880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830314.679928:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830314.679931:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830314.679933:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830314.679937:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6765 00000100:00080000:0.0:1787830314.679940:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830314.679944:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830314.679948:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830314.679958:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973290752:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830314.679961:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973291008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830314.679974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830314.679977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830314.679978:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830314.679982:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6765 00000100:00080000:0.0:1787830314.679985:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830314.679989:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830314.679993:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830314.680003:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973291136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830314.680011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830314.680014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1787830314.680024:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830314.680028:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830314.680033:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6765 00000100:00080000:0.0:1787830314.680038:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830314.680042:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830314.680046:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830314.680058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973291264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830314.680066:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830314.680068:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830314/real 1787830314] req@ffff9c1a78863840 x1874668973290752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830314.680071:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830314.680074:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830314.680081:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6765 00000100:00080000:0.0:1787830314.680087:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830314.680091:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830314.680096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973290752/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787830314.680097:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787830314.680111:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787830314.680112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973291392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830314.680120:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787830314.680127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830314.680128:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973290752:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787830314.680136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830314.680139:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830314.680148:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6765 00000100:00080000:0.0:1787830314.680151:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830314.680155:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830314.680155:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973290880:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787830314.680163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830314.680177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973291520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830314.680191:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830314.680210:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830314/real 1787830314] req@ffff9c1a78860000 x1874668973290880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830314.680224:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973290880/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830314.680233:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830314.680237:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830314.680240:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973290880:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830314.680264:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973291008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830314.680285:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830314.680296:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830314/real 1787830314] req@ffff9c1a788630c0 x1874668973291008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830314.680308:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973291008/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830314.680316:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830314.680319:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830314.680322:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973291008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830314.680329:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973291136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830314.680388:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830314.680401:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830314/real 1787830314] req@ffff9c1a78862580 x1874668973291136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830314.680413:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973291136/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830314.680422:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830314.680425:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830314.680428:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973291136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830314.680433:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973291264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830314.680453:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830314.680464:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830314/real 1787830314] req@ffff9c1a78861680 x1874668973291264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830314.680475:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973291264/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830314.680483:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830314.680486:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830314.680489:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973291264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830314.680494:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973291392:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830314.680514:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830314.680525:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830314/real 1787830314] req@ffff9c1a78861e00 x1874668973291392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830314.680537:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973291392/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830314.680544:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830314.680547:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830314.680550:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973291392:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830314.680555:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973291520:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830314.680572:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830314.680582:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830314/real 1787830314] req@ffff9c1a788612c0 x1874668973291520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830314.680594:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973291520/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830314.680607:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830314.680610:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830314.680614:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973291520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830315.191468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830315.319732:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830315.512132:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830316.215517:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830316.343510:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830316.535543:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830317.239523:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830317.367570:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830317.559595:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830318.263594:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830318.391448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830318.583599:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830319.287645:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830319.415849:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830319.607563:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830319.800185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830319.800193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830319.800197:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830319.800207:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6770 00000100:00080000:0.0:1787830319.800217:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830319.800223:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830319.800263:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973291648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830319.800288:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830319.800291:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830319.800293:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830319.800297:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6770 00000100:00080000:0.0:1787830319.800300:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830319.800304:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830319.800313:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830319.800326:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973291776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830319.800335:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830319.800338:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830319.800339:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830319.800342:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6770 00000100:00080000:0.0:1787830319.800364:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830319.800368:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830319.800373:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830319.800384:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973291904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830319.800395:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830319.800398:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830319.800399:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830319.800402:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6770 00000100:00080000:0.0:1787830319.800406:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830319.800409:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830319.800414:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830319.800423:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973292032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830319.800431:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830319.800434:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830319.800436:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830319.800438:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6770 00000100:00080000:0.0:1787830319.800441:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830319.800824:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830319.800831:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830319.800844:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973292160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830319.800859:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830319.800862:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830319.800863:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830319.800867:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6770 00000100:00080000:0.0:1787830319.800871:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830319.800875:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830319.800879:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830319.800887:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973292288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830319.800895:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830319.800898:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830319.800899:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830319.800903:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6770 00000100:00080000:0.0:1787830319.800906:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830319.800909:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830319.800913:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830319.800921:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973292416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830319.800995:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973291648:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830319.801050:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830319.801083:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830319/real 1787830319] req@ffff9c1a78863c00 x1874668973291648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830319.801098:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973291648/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830319.801108:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830319.801112:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830319.801115:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973291648:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830319.801122:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973291776:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830319.801144:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830319.801156:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830319/real 1787830319] req@ffff9c1a788612c0 x1874668973291776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830319.801168:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973291776/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830319.801176:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830319.801179:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830319.801182:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973291776:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830319.801189:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973291904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830319.801208:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830319.801219:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830319/real 1787830319] req@ffff9c1a78861e00 x1874668973291904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830319.801231:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973291904/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830319.801238:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830319.801242:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830319.801245:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973291904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830319.801250:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973292032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830319.801380:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830319.801395:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830319/real 1787830319] req@ffff9c1a78861680 x1874668973292032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830319.801410:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973292032/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830319.801418:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830319.801423:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830319.801426:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973292032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830319.801433:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973292160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830319.801597:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830319.801609:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830319/real 1787830319] req@ffff9c1a78862580 x1874668973292160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830319.801621:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973292160/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830319.801639:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830319.801643:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830319.801646:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973292160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830319.801652:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973292288:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830319.801672:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830319.801683:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830319/real 1787830319] req@ffff9c1a788630c0 x1874668973292288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830319.801695:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973292288/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830319.801702:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830319.801705:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830319.801708:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973292288:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830319.801714:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973292416:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830319.801732:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830319.801743:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830319/real 1787830319] req@ffff9c1a78860000 x1874668973292416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830319.801754:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973292416/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830319.801768:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830319.801771:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830319.801774:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973292416:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830320.311441:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830320.440843:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830320.632211:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830321.335463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830321.464176:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830321.656515:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830322.359587:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830322.488204:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830322.679870:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830323.383467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830323.511460:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830323.703786:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830324.407604:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830324.535643:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830324.728171:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830324.919523:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830324.919534:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830324.919538:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830324.919549:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6775 00000100:00080000:3.0:1787830324.919559:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830324.919566:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830324.919614:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973292544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830324.919631:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830324.919635:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830324.919636:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830324.919639:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6775 00000100:00080000:3.0:1787830324.919642:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830324.919644:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830324.919651:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830324.919660:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973292672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830324.919666:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830324.919668:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830324.919669:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830324.919671:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6775 00000100:00080000:3.0:1787830324.919672:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830324.919674:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830324.919676:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830324.919682:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973292800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830324.919688:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830324.919690:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830324.919691:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830324.919693:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6775 00000100:00080000:3.0:1787830324.919695:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830324.919697:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830324.919699:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830324.919705:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973292928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830324.919710:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830324.919711:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830324.919712:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830324.919714:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6775 00000100:00080000:3.0:1787830324.919715:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830324.919717:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830324.919719:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830324.919724:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973293056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830324.919728:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830324.919730:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830324.919731:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830324.919733:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6775 00000100:00080000:3.0:1787830324.919735:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830324.919737:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830324.919739:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830324.919744:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973293184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830324.919748:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830324.919750:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830324.919751:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830324.919753:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6775 00000100:00080000:3.0:1787830324.919755:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830324.919757:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830324.919759:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830324.919764:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973293312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830324.920072:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973292544:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830324.920126:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830324.920160:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830324/real 1787830324] req@ffff9c1a7895d2c0 x1874668973292544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830324.920177:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973292544/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830324.920184:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830324.920187:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830324.920189:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973292544:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830324.920194:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973292672:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830324.920210:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830324.920400:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830324/real 1787830324] req@ffff9c1a7895f480 x1874668973292672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830324.920414:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973292672/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830324.920422:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830324.920425:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830324.920428:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973292672:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830324.920435:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973292800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830324.920457:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830324.920468:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830324/real 1787830324] req@ffff9c1a7895c000 x1874668973292800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830324.920480:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973292800/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830324.920487:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830324.920490:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830324.920493:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973292800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830324.920499:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973292928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830324.920517:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830324.920526:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830324/real 1787830324] req@ffff9c1a7895e580 x1874668973292928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830324.920536:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973292928/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830324.920543:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830324.920546:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830324.920548:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973292928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830324.920553:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973293056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830324.920569:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830324.920578:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830324/real 1787830324] req@ffff9c1a7895c780 x1874668973293056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830324.920589:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973293056/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830324.920610:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830324.920613:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830324.920616:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973293056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830324.920622:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973293184:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830324.920641:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830324.920651:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830324/real 1787830324] req@ffff9c1a7895cf00 x1874668973293184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830324.920662:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973293184/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830324.920669:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830324.920672:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830324.920674:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973293184:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830324.920679:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973293312:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830324.920697:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830324.920705:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830324/real 1787830324] req@ffff9c1a7895e940 x1874668973293312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830324.920714:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973293312/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830324.920726:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830324.920729:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830324.920732:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973293312:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830325.431461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830325.559522:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830325.751523:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830326.455423:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830326.583724:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830326.775548:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830327.479463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830327.607432:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830327.799606:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830328.503470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830328.631540:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830328.824165:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830329.527452:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830329.656546:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830329.847782:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830330.039592:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830330.039601:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830330.039606:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830330.039616:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6780 00000100:00080000:0.0:1787830330.039627:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830330.039635:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830330.039746:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973293440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830330.039774:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830330.039778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830330.039779:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830330.039783:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6780 00000100:00080000:0.0:1787830330.039788:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830330.039793:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830330.039802:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830330.039813:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973293568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830330.039822:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830330.039825:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830330.039827:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830330.039830:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6780 00000100:00080000:0.0:1787830330.039833:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830330.039836:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830330.039841:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830330.039850:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973293696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830330.039863:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830330.039865:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830330.039867:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830330.039870:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6780 00000100:00080000:0.0:1787830330.039873:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830330.039876:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830330.039879:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830330.039890:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973293824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830330.039908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830330.039910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830330.039912:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830330.039915:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6780 00000100:00080000:0.0:1787830330.039917:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830330.039921:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830330.039925:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830330.040002:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973293952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830330.040012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830330.040014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830330.040016:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830330.040019:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6780 00000100:00080000:0.0:1787830330.040023:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830330.040026:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830330.040032:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830330.040041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973294080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830330.040049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830330.040052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830330.040053:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830330.040057:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6780 00000100:00080000:0.0:1787830330.040060:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830330.040063:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830330.040067:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830330.040075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973294208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830330.041669:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973293440:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830330.041729:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830330.041764:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830330/real 1787830330] req@ffff9c1a78863840 x1874668973293440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830330.041780:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973293440/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830330.041790:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830330.041794:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830330.041798:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973293440:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830330.041805:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973293568:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830330.041824:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830330.041834:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830330/real 1787830330] req@ffff9c1a78861a40 x1874668973293568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830330.041845:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973293568/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830330.041853:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830330.041856:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830330.041858:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973293568:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830330.041865:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973293696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830330.041880:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830330.041888:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830330/real 1787830330] req@ffff9c1a78860b40 x1874668973293696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830330.041898:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973293696/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830330.041904:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830330.041907:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830330.041910:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973293696:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830330.041915:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973293824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830330.041931:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830330.041940:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830330/real 1787830330] req@ffff9c1a78862d00 x1874668973293824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830330.041950:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973293824/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830330.041958:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830330.041961:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830330.041964:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973293824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830330.041969:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973293952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830330.041983:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830330.041991:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830330/real 1787830330] req@ffff9c1a78860780 x1874668973293952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830330.042000:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973293952/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830330.042007:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830330.042009:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830330.042012:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973293952:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830330.042017:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973294080:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830330.042030:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830330.042038:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830330/real 1787830330] req@ffff9c1a78862940 x1874668973294080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830330.042047:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973294080/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830330.042054:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830330.042057:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830330.042060:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973294080:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830330.042065:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973294208:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830330.042081:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830330.042091:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830330/real 1787830330] req@ffff9c1a788621c0 x1874668973294208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830330.042101:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973294208/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830330.042113:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830330.042116:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830330.042119:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973294208:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830330.551401:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830330.679616:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830330.871601:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830331.575501:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830331.703594:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830331.895634:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830332.599493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830332.727503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830332.919829:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830333.623509:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830333.751899:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830333.943917:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830334.647648:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830334.775556:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830334.968540:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830335.159586:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830335.159599:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830335.159605:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830335.159618:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6785 00000100:00080000:2.0:1787830335.159632:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830335.159640:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830335.159691:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973294336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830335.159790:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973294336:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787830335.159873:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787830335.159911:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830335.159915:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830335.159917:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830335.159923:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6785 00000100:00080000:2.0:1787830335.159927:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830335.159932:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830335.159944:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830335.159959:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973294464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830335.160227:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830335/real 1787830335] req@ffff9c1a49d8e1c0 x1874668973294336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830335.160246:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973294336/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830335.160264:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830335.160268:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830335.160272:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973294336:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830335.160420:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973294464:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830335.160450:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830335.160463:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830335/real 1787830335] req@ffff9c1a49d8f0c0 x1874668973294464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830335.160973:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830335.160978:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830335.161088:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830335.161100:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6785 00000100:00080000:2.0:1787830335.161112:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830335.161119:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830335.161133:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830335.161167:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973294592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830335.161185:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830335.161189:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830335.161191:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830335.161194:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6785 00000100:00080000:2.0:1787830335.161203:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830335.161208:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830335.161212:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830335.161258:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973294720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830335.161269:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830335.161272:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830335.161274:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830335.161278:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6785 00000100:00080000:2.0:1787830335.161282:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830335.161286:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830335.161291:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830335.161302:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973294848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830335.161313:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973294464/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830335.161313:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830335.161324:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830335.161327:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830335.161374:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830335.161378:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6785 00000100:00080000:0.0:1787830335.161381:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787830335.161387:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830335.161389:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973294464:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787830335.161396:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830335.161405:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830335.161422:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973294592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830335.161424:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973294976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830335.161451:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830335.161455:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830335.161456:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830335.161461:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6785 00000100:00080000:2.0:1787830335.161467:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787830335.161469:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:2.0:1787830335.161478:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830335.161485:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830335.161493:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830335/real 1787830335] req@ffff9c1a49d8c3c0 x1874668973294592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830335.161503:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973295104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830335.161516:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973294592/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830335.161535:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830335.161538:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830335.161542:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973294592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830335.161549:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973294720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830335.161569:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830335.161578:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830335/real 1787830335] req@ffff9c1a49d8e580 x1874668973294720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830335.161590:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973294720/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830335.161598:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830335.161605:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830335.161609:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973294720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830335.161614:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973294848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830335.161629:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830335.161638:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830335/real 1787830335] req@ffff9c1a49d8d680 x1874668973294848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830335.161648:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973294848/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830335.161695:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830335.161699:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830335.161705:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973294848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830335.161727:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973294976:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830335.161761:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830335.161786:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830335/real 1787830335] req@ffff9c1a49d8de00 x1874668973294976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830335.161810:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973294976/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830335.161823:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830335.161828:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830335.161833:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973294976:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830335.161844:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973295104:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830335.161872:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830335.161896:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830335/real 1787830335] req@ffff9c1a49d8c780 x1874668973295104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830335.161911:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973295104/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830335.161919:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830335.161923:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830335.161926:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973295104:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830335.671552:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830335.799604:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830335.991549:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830336.695687:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830336.823532:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830337.015613:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830337.719473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830337.847644:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830338.039886:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830338.743469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830338.871472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830339.063553:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830339.767465:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830339.895514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830340.087878:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830340.279557:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830340.279567:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830340.279571:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830340.279582:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6790 00000100:00080000:2.0:1787830340.279596:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830340.279604:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830340.279642:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973295232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830340.279670:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830340.279673:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830340.279675:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830340.279678:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6790 00000100:00080000:2.0:1787830340.279681:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830340.279684:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830340.279691:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830340.279701:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973295360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830340.279707:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830340.279709:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830340.279710:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830340.279712:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6790 00000100:00080000:2.0:1787830340.279714:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830340.279717:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830340.279720:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830340.279728:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973295488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830340.279743:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830340.279746:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830340.279747:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830340.279751:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6790 00000100:00080000:2.0:1787830340.279754:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830340.279758:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830340.279762:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830340.279773:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973295616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830340.279782:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830340.279786:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830340.279787:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830340.279791:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6790 00000100:00080000:2.0:1787830340.279795:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830340.279797:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973295232:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787830340.279801:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830340.279809:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830340.279821:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973295744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830340.279830:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830340.279833:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830340.279836:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830340.279839:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6790 00000100:00080000:2.0:1787830340.279844:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830340.279847:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830340.279852:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787830340.279866:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830340.279866:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973295872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830340.279878:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830340.279882:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830340.279884:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830340.279888:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6790 00000100:00080000:2.0:1787830340.279892:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830340.279895:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830340.279899:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830340.279910:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973296000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830340.280034:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830340/real 1787830340] req@ffff9c1a49d8c000 x1874668973295232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830340.280053:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973295232/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830340.280063:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830340.280067:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830340.280071:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973295232:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830340.280077:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973295360:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830340.280100:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830340.280111:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830340/real 1787830340] req@ffff9c1a49d8cf00 x1874668973295360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830340.280121:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973295360/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830340.280128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830340.280132:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830340.280135:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973295360:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830340.280140:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973295488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830340.280155:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830340.280164:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830340/real 1787830340] req@ffff9c1a49d8da40 x1874668973295488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830340.280173:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973295488/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830340.280179:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830340.280181:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830340.280184:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973295488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830340.280189:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973295616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830340.280205:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830340.280215:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830340/real 1787830340] req@ffff9c1a49d8cb40 x1874668973295616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830340.280226:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973295616/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830340.280233:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830340.280235:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830340.280237:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973295616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830340.280262:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973295744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830340.280279:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830340.280289:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830340/real 1787830340] req@ffff9c1a49d8c780 x1874668973295744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830340.280299:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973295744/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830340.280306:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830340.280308:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830340.280311:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973295744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830340.280316:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973295872:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830340.280334:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830340.280343:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830340/real 1787830340] req@ffff9c1a49d8de00 x1874668973295872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830340.280392:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973295872/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830340.280400:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830340.280403:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830340.280405:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973295872:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830340.280411:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973296000:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830340.280428:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830340.280437:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830340/real 1787830340] req@ffff9c1a49d8d680 x1874668973296000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830340.280448:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973296000/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830340.280455:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830340.280459:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830340.280461:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973296000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830340.791652:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830340.919472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830341.112112:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830341.815489:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830341.943598:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830342.136317:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830342.839508:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830342.967454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830343.159560:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830343.864582:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830343.991448:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830344.183630:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830344.887835:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830345.015475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830345.207548:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830345.401592:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830345.401603:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830345.401608:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830345.401619:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6795 00000100:00080000:0.0:1787830345.401630:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830345.401637:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830345.401675:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973296128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830345.401706:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830345.401710:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830345.401712:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830345.401716:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6795 00000100:00080000:0.0:1787830345.401719:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830345.401723:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830345.401735:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830345.401745:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973296128:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830345.401756:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973296256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830345.401766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830345.401769:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830345.401771:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830345.401774:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6795 00000100:00080000:0.0:1787830345.401777:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830345.401780:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830345.401784:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830345.401794:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973296384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830345.401802:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830345.401805:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830345.401807:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830345.401811:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6795 00000100:00080000:0.0:1787830345.401814:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830345.401818:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830345.401823:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830345.401835:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973296512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830345.401844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830345.401847:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830345.401848:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830345.401852:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6795 00000800:00000100:2.0:1787830345.401856:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830345.401858:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830345.401865:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830345.401870:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830345.401882:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973296640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830345.401891:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830345.401896:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787830345.401897:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830345/real 1787830345] req@ffff9c1a78860f00 x1874668973296128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830345.401899:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830345.401908:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6795 00000100:00080000:0.0:1787830345.401915:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830345.401920:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830345.401925:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830345.401925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973296128/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830345.401940:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787830345.401942:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973296768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830345.401948:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830345.401956:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973296128:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787830345.401962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830345.401967:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830345.401969:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830345.401974:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6795 00000100:00080000:0.0:1787830345.401978:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830345.401984:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973296256:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830345.401985:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830345.401993:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830345.402007:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973296896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830345.402016:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830345.402032:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830345/real 1787830345] req@ffff9c1a788621c0 x1874668973296256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830345.402044:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973296256/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830345.402051:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830345.402054:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830345.402057:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973296256:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830345.402062:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973296384:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830345.402077:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830345.402085:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830345/real 1787830345] req@ffff9c1a78862940 x1874668973296384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830345.402096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973296384/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830345.402103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830345.402106:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830345.402109:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973296384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830345.402115:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973296512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830345.402130:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830345.402138:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830345/real 1787830345] req@ffff9c1a78860780 x1874668973296512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830345.402148:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973296512/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830345.402157:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830345.402160:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830345.402163:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973296512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830345.402168:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973296640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830345.402182:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830345.402190:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830345/real 1787830345] req@ffff9c1a78862d00 x1874668973296640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830345.402201:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973296640/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830345.402208:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830345.402211:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830345.402214:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973296640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830345.402220:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973296768:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830345.402236:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830345.402246:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830345/real 1787830345] req@ffff9c1a78860b40 x1874668973296768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830345.402258:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973296768/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830345.402266:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830345.402268:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830345.402271:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973296768:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830345.402287:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973296896:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830345.402303:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830345.402311:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830345/real 1787830345] req@ffff9c1a78861a40 x1874668973296896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830345.402321:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973296896/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830345.402328:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830345.402331:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830345.402334:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973296896:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830345.911673:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830346.039475:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830346.231661:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830346.935726:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830347.063468:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830347.255661:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830347.959651:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830348.087470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830348.279552:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830348.983479:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830349.111470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830349.303514:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830350.007671:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830350.135480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830350.327950:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830350.519522:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830350.519532:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830350.519537:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830350.519547:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6800 00000100:00080000:0.0:1787830350.519557:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830350.519564:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830350.519603:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973297024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830350.519626:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830350.519629:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830350.519632:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830350.519637:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6800 00000100:00080000:0.0:1787830350.519640:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830350.519644:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830350.519653:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830350.519664:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973297152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830350.519672:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830350.519675:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830350.519677:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830350.519680:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6800 00000100:00080000:0.0:1787830350.519683:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830350.519686:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830350.519691:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830350.519700:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973297280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830350.519710:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830350.519712:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830350.519714:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830350.519717:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6800 00000100:00080000:0.0:1787830350.519719:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830350.519722:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830350.519726:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830350.519735:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973297408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830350.519746:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830350.519749:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830350.519751:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830350.519753:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973297024:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787830350.519769:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6800 00000100:00080000:0.0:1787830350.519774:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830350.519778:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830350.519784:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830350.519798:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973297536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830350.519808:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830350.519811:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830350.519816:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787830350.519816:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830350.519824:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6800 00000100:00080000:0.0:1787830350.519828:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830350.519832:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830350.519837:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830350.519845:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973297664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830350.519854:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830350.519856:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830350.519858:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830350.519861:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6800 00000100:00080000:0.0:1787830350.519864:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830350.519868:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830350.519872:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830350.519883:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973297792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830350.519906:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830350/real 1787830350] req@ffff9c1a78863840 x1874668973297024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830350.519925:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973297024/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830350.519932:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830350.519935:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830350.519937:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973297024:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830350.519943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973297152:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830350.519960:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830350.519968:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830350/real 1787830350] req@ffff9c1a78860000 x1874668973297152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830350.519975:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973297152/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830350.519980:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830350.519982:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830350.519984:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973297152:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830350.519987:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973297280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830350.519996:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830350.520001:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830350/real 1787830350] req@ffff9c1a788630c0 x1874668973297280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830350.520008:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973297280/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830350.520016:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830350.520018:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830350.520019:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973297280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830350.520038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973297408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830350.520048:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830350.520053:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830350/real 1787830350] req@ffff9c1a78862580 x1874668973297408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830350.520059:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973297408/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830350.520064:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830350.520065:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830350.520067:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973297408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830350.520070:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973297536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830350.520078:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830350.520083:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830350/real 1787830350] req@ffff9c1a78861680 x1874668973297536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830350.520091:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973297536/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830350.520098:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830350.520101:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830350.520104:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973297536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830350.520109:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973297664:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830350.520137:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830350.520148:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830350/real 1787830350] req@ffff9c1a78861e00 x1874668973297664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830350.520157:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973297664/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830350.520162:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830350.520164:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830350.520165:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973297664:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830350.520169:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973297792:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830350.520181:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830350.520186:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830350/real 1787830350] req@ffff9c1a788612c0 x1874668973297792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830350.520192:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973297792/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830350.520197:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830350.520198:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830350.520200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973297792:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830351.031477:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830351.159455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830351.351607:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830352.055521:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830352.183567:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830352.376457:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830353.079488:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830353.207575:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830353.399572:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830354.103487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830354.231473:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830354.425104:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830355.127490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830355.255607:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830355.447669:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830355.639741:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830355.639749:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830355.639753:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830355.639762:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6805 00000100:00080000:0.0:1787830355.639771:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830355.639779:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830355.639872:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973297920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830355.639902:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830355.639905:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830355.639907:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830355.640278:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6805 00000100:00080000:0.0:1787830355.640284:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830355.640289:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830355.640298:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830355.640312:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973298048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830355.640328:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830355.640330:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830355.640332:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830355.640336:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6805 00000100:00080000:0.0:1787830355.640339:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830355.640343:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830355.640478:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973297920:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830355.640534:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830355.640567:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830355/real 1787830355] req@ffff9c1a78863c00 x1874668973297920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830355.640583:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973297920/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830355.640598:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830355.640602:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830355.640606:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973297920:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830355.640613:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973298048:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787830355.640628:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787830355.640633:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830355.640641:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973298176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830355.640644:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830355/real 1787830355] req@ffff9c1a788612c0 x1874668973298048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830355.640652:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830355.640656:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787830355.640656:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973298048/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830355.640658:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830355.640661:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6805 00000100:00080000:3.0:1787830355.640663:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830355.640665:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830355.640667:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787830355.640669:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830355.640669:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973298048:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787830355.640673:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830355.640681:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973298304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830355.640689:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830355.640691:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973298176:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830355.640692:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830355.640693:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830355.640697:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6805 00000100:00080000:0.0:1787830355.640700:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830355.640703:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830355.640707:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787830355.640707:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830355.640716:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973298432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830355.640717:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830355/real 1787830355] req@ffff9c1a78861e00 x1874668973298176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830355.640725:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830355.640728:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787830355.640728:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973298176/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830355.640730:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830355.640733:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6805 00000100:00080000:3.0:1787830355.640735:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830355.640736:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830355.640738:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787830355.640739:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830355.640741:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973298176:192.168.203.148@tcp:38:kworker.0 00000008:00100000:0.0:1787830355.640744:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830355.640747:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973298304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830355.640752:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973298560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830355.640760:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787830355.640761:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830355.640763:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830355.640764:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830355.640767:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6805 00000100:00080000:0.0:1787830355.640770:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830355.640770:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830355/real 1787830355] req@ffff9c1a78861680 x1874668973298304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830355.640773:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830355.640777:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830355.640781:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973298304/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830355.640785:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973298688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830355.640789:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830355.640792:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830355.640795:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973298304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830355.640806:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973298432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830355.640823:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830355.640833:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830355/real 1787830355] req@ffff9c1a78862580 x1874668973298432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830355.640844:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973298432/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830355.640851:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830355.640854:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830355.640857:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973298432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830355.640862:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973298560:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830355.640881:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830355.640891:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830355/real 1787830355] req@ffff9c1a788630c0 x1874668973298560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830355.640902:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973298560/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830355.640910:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830355.640913:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830355.640915:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973298560:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830355.640921:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973298688:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830355.640938:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830355.640948:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830355/real 1787830355] req@ffff9c1a78860000 x1874668973298688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830355.640959:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973298688/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830355.640966:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830355.640970:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830355.640972:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973298688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830356.151487:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830356.279486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830356.471506:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830357.175599:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830357.303550:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830357.495838:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830358.199480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830358.327480:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830358.519604:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830359.223492:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830359.352751:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830359.545542:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830360.249014:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830360.376026:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830360.567543:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830360.759466:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830360.759477:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830360.759481:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830360.759492:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6811 00000100:00080000:0.0:1787830360.759502:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830360.759510:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830360.759547:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973298816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830360.759621:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973298816:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830360.759680:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830360.759712:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830360.759716:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830360.759718:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830360.759722:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6811 00000100:00080000:0.0:1787830360.759727:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830360.759731:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830360.759739:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830360.759752:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973298944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830360.759808:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830360/real 1787830360] req@ffff9c1a78863840 x1874668973298816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830360.759824:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973298816/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830360.759833:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830360.759837:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830360.759841:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973298816:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830360.759863:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973298944:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830360.759886:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830360.759899:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830360/real 1787830360] req@ffff9c1a78861a40 x1874668973298944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830360.759910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973298944/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830360.759918:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830360.759921:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830360.759924:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973298944:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830360.760018:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830360.760021:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830360.760023:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830360.760027:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6811 00000100:00080000:0.0:1787830360.760031:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830360.760035:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830360.760039:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830360.760050:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973299072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830360.760091:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973299072:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830360.760111:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830360.760131:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830360.760135:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830360.760137:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830360.760140:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6811 00000100:00080000:0.0:1787830360.760144:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830360.760147:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830360.760151:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830360.760164:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973299200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830360.760203:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830360/real 1787830360] req@ffff9c1a78860b40 x1874668973299072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830360.760220:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973299072/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830360.760228:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830360.760231:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830360.760234:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973299072:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830360.760245:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973299200:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830360.760265:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830360.760275:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830360/real 1787830360] req@ffff9c1a78862d00 x1874668973299200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830360.760286:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973299200/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830360.760293:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830360.760297:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830360.760300:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973299200:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830360.760412:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830360.760416:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830360.760418:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830360.760422:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6811 00000100:00080000:0.0:1787830360.760426:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830360.760430:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830360.760435:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830360.760447:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973299328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830360.760491:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973299328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830360.760514:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830360.760535:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830360.760538:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830360.760540:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830360.760544:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6811 00000100:00080000:0.0:1787830360.760548:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830360.760552:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830360.760557:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830360.760568:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973299456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830360.760604:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830360/real 1787830360] req@ffff9c1a78860780 x1874668973299328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830360.760616:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973299328/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830360.760628:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830360.760632:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830360.760635:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973299328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830360.760646:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973299456:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830360.760666:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830360.760677:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830360/real 1787830360] req@ffff9c1a78862940 x1874668973299456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830360.760688:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973299456/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830360.760695:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830360.760698:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830360.760702:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973299456:192.168.203.148@tcp:8:kworker.0 00000100:00080000:0.0:1787830360.760780:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830360.760784:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830360.760786:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830360.760790:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6811 00000100:00080000:0.0:1787830360.760794:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830360.760797:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830360.760802:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830360.760813:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973299584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830360.760852:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973299584:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830360.760872:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830360.760936:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830360/real 1787830360] req@ffff9c1a788621c0 x1874668973299584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830360.760948:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973299584/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830360.760956:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830360.760959:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830360.760962:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973299584:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830361.272714:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830361.399528:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830361.592388:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830362.295468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830362.423470:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830362.615486:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830363.319450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830363.447525:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830363.639554:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830364.343447:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830364.471455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830364.663555:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830365.367490:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830365.495501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830365.687557:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830365.880285:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830365.880302:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830365.880314:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830365.880324:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6816 00000100:00080000:2.0:1787830365.880334:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830365.880340:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830365.880391:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973299712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830365.880427:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830365.880430:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830365.880432:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830365.880435:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6816 00000100:00080000:2.0:1787830365.880439:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830365.880442:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830365.880456:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830365.880467:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973299840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830365.880476:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830365.880479:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830365.880481:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830365.880485:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6816 00000100:00080000:2.0:1787830365.880487:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830365.880490:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830365.880494:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830365.880501:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973299968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830365.880510:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830365.880514:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830365.880515:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830365.880518:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6816 00000100:00080000:2.0:1787830365.880521:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830365.880524:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830365.880528:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830365.880535:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973300096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830365.880544:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830365.880547:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830365.880549:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830365.880554:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6816 00000100:00100000:3.0:1787830365.880557:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973299712:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787830365.880558:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830365.880565:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830365.880574:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830365.880584:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973300224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830365.880593:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830365.880596:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830365.880598:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830365.880602:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6816 00000100:00080000:2.0:1787830365.880614:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830365.880617:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830365.880622:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830365.880630:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973300352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830365.880639:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830365.880649:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830365.880651:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830365.880655:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973299840:192.168.203.148@tcp:38:kworker.0 00000100:00080000:2.0:1787830365.880659:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6816 00000100:00080000:2.0:1787830365.880665:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830365.880669:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830365.880675:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830365.880687:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973300480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830365.880694:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973299968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830365.880725:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973300096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830365.880755:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973300224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830365.880777:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973300352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830365.880798:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973300480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830366.391523:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830366.519465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830366.711548:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830367.415514:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830367.543466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830367.735764:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830368.439472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830368.568224:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830368.759814:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787830368.951638:0:2399:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787830368.951680:0:2399:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830368.951695:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787830368.951784:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830365/real 1787830368] req@ffff9c1a49d8e580 x1874668973299712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830368.951808:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973299712/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830368.951819:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830368.951824:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830368.951828:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973299712:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830368.951878:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830368.951896:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830368.951904:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830368.951911:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830368.951919:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830368.951927:0:2399:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787830368.953576:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830365/real 1787830368] req@ffff9c1a49d8ed00 x1874668973300224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830368.953603:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973300224/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830368.953614:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830368.953618:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830368.953622:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973300224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830368.953639:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830365/real 1787830368] req@ffff9c1a49d8fc00 x1874668973300352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830368.953651:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973300352/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830368.953659:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830368.953663:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830368.953666:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973300352:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830368.953673:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830365/real 1787830368] req@ffff9c1a49d8d680 x1874668973300480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830368.953683:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973300480/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830368.953692:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830368.953695:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830368.953698:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973300480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830368.953704:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830365/real 1787830368] req@ffff9c1a49d8c3c0 x1874668973299840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830368.953716:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973299840/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830368.953723:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830368.953726:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830368.953728:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973299840:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830368.953734:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830365/real 1787830368] req@ffff9c1a49d8f0c0 x1874668973299968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830368.953744:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973299968/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830368.953751:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830368.953754:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830368.953757:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973299968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830368.953764:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830365/real 1787830368] req@ffff9c1a49d8e1c0 x1874668973300096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830368.953773:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973300096/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830368.953780:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830368.953784:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830368.953786:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973300096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830369.463470:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830369.591454:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830369.786140:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830370.487481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830370.615493:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830370.807889:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830371.001522:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830371.001532:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830371.001536:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830371.001546:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6821 00000100:00080000:1.0:1787830371.001557:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830371.001565:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830371.001604:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973300608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830371.001631:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830371.001635:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830371.001636:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830371.001641:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6821 00000100:00080000:1.0:1787830371.001645:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830371.001648:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830371.001657:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830371.001669:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973300736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830371.001677:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830371.001679:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830371.001681:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830371.001684:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6821 00000100:00080000:1.0:1787830371.001687:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830371.001691:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830371.001695:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830371.001705:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973300864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830371.001713:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830371.001716:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830371.001717:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830371.001720:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6821 00000100:00080000:1.0:1787830371.001722:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830371.001725:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830371.001729:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830371.001749:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973300992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830371.001759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830371.001761:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830371.001763:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830371.001766:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6821 00000100:00080000:1.0:1787830371.001769:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830371.001772:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830371.001776:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830371.001785:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973301120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830371.001793:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830371.001796:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830371.001797:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830371.001800:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6821 00000100:00080000:1.0:1787830371.001803:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830371.001806:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830371.001810:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830371.001819:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973301248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830371.001827:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830371.001830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830371.001831:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830371.001834:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6821 00000100:00080000:1.0:1787830371.001837:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830371.001840:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830371.001844:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830371.001853:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973301376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830371.002502:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973300608:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830371.002601:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830371.002652:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830371/real 1787830371] req@ffff9c1a78a7b0c0 x1874668973300608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830371.002671:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973300608/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830371.002682:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830371.002688:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830371.002692:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973300608:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830371.002701:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973300736:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830371.002724:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830371.002735:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830371/real 1787830371] req@ffff9c1a78a7a1c0 x1874668973300736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830371.002746:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973300736/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830371.002755:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830371.002759:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830371.002762:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973300736:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830371.002769:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973300864:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830371.002785:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830371.002796:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830371/real 1787830371] req@ffff9c1a78a783c0 x1874668973300864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830371.002807:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973300864/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830371.002813:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830371.002816:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830371.002819:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973300864:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830371.002825:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973300992:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830371.002844:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830371.002855:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830371/real 1787830371] req@ffff9c1a78a79e00 x1874668973300992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830371.002866:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973300992/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830371.002875:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830371.002878:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830371.002881:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973300992:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830371.002887:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973301120:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830371.002904:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830371.002912:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830371/real 1787830371] req@ffff9c1a78a7a940 x1874668973301120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830371.002923:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973301120/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830371.002931:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830371.002934:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830371.002937:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973301120:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830371.002942:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973301248:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830371.002957:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830371.002965:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830371/real 1787830371] req@ffff9c1a78a78780 x1874668973301248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830371.002976:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973301248/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830371.002984:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830371.002987:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830371.002990:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973301248:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830371.002996:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973301376:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830371.003009:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830371.003016:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830371/real 1787830371] req@ffff9c1a78a7bc00 x1874668973301376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830371.003034:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973301376/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830371.003041:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830371.003044:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830371.003047:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973301376:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830371.511485:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830371.639488:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830371.831600:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830372.536722:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830372.663456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830372.855598:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830373.560270:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830373.688702:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830373.879582:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830374.583616:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830374.711499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830374.903481:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830375.607919:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830375.735677:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830375.929770:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830376.119503:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830376.119512:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830376.119517:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830376.119526:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6826 00000100:00080000:3.0:1787830376.119536:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830376.119542:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830376.119575:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c3c0 x1874668973301504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830376.119604:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830376.119607:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830376.119608:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830376.119612:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6826 00000100:00080000:3.0:1787830376.119616:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830376.119620:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830376.119628:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830376.119640:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e940 x1874668973301632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830376.119649:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830376.119651:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830376.119653:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830376.119656:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6826 00000100:00080000:3.0:1787830376.119659:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830376.119662:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830376.119666:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830376.119673:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cf00 x1874668973301760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830376.119682:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830376.119684:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830376.119686:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830376.119689:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6826 00000100:00080000:3.0:1787830376.119692:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830376.119695:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830376.119698:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830376.119706:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c780 x1874668973301888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830376.119713:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830376.119717:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830376.119719:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830376.119722:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6826 00000100:00080000:3.0:1787830376.119725:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830376.119728:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830376.119731:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830376.119738:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e580 x1874668973302016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830376.119746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830376.119749:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830376.119750:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830376.119754:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6826 00000100:00080000:3.0:1787830376.119756:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830376.119760:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830376.119765:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830376.119773:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895c000 x1874668973302144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830376.119780:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830376.119783:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830376.119784:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830376.119787:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6826 00000100:00080000:3.0:1787830376.119790:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830376.119794:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830376.119798:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830376.119806:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f480 x1874668973302272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830376.119841:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973301504:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830376.119915:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830376.119952:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830376/real 1787830376] req@ffff9c1a7895c3c0 x1874668973301504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830376.119968:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c3c0 x1874668973301504/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830376.119976:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830376.119980:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830376.119983:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973301504:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830376.119989:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973301632:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830376.120009:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830376.120020:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830376/real 1787830376] req@ffff9c1a7895e940 x1874668973301632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830376.120030:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e940 x1874668973301632/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830376.120038:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830376.120042:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830376.120046:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973301632:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830376.120052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973301760:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830376.120070:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830376.120080:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830376/real 1787830376] req@ffff9c1a7895cf00 x1874668973301760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830376.120090:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cf00 x1874668973301760/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830376.120096:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830376.120099:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830376.120101:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973301760:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830376.120106:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973301888:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830376.120122:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830376.120140:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830376/real 1787830376] req@ffff9c1a7895c780 x1874668973301888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830376.120151:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c780 x1874668973301888/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830376.120157:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830376.120160:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830376.120163:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973301888:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830376.120169:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973302016:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830376.120186:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830376.120197:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830376/real 1787830376] req@ffff9c1a7895e580 x1874668973302016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830376.120207:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e580 x1874668973302016/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830376.120214:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830376.120217:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830376.120219:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973302016:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830376.120225:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973302144:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830376.120254:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830376.120265:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830376/real 1787830376] req@ffff9c1a7895c000 x1874668973302144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830376.120277:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895c000 x1874668973302144/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830376.120284:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830376.120286:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830376.120289:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973302144:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830376.120295:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973302272:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830376.120313:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830376.120323:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830376/real 1787830376] req@ffff9c1a7895f480 x1874668973302272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830376.120334:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f480 x1874668973302272/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830376.120342:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830376.120366:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830376.120369:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973302272:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830376.631950:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830376.759463:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830376.951639:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830377.655466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830377.783692:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830377.975547:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830378.679504:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830378.807467:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830379.000010:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830379.703463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830379.831461:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830380.023539:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830380.728511:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830380.855451:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830381.047781:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830381.239544:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830381.239554:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830381.239558:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830381.239566:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6831 00000100:00080000:0.0:1787830381.239576:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830381.239584:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830381.239625:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973302400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830381.239649:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830381.239653:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830381.239654:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830381.239658:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6831 00000100:00080000:0.0:1787830381.239662:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830381.239666:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830381.239675:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830381.239800:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973302528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830381.239819:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830381.239822:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830381.239824:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830381.239829:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6831 00000100:00080000:0.0:1787830381.239833:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830381.239837:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830381.239843:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830381.239845:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973302400:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830381.239858:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973302656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830381.239869:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830381.239873:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830381.239875:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830381.239879:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6831 00000100:00080000:0.0:1787830381.239884:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830381.239888:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:2.0:1787830381.239907:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830381.239946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830381/real 1787830381] req@ffff9c1a78860f00 x1874668973302400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787830381.239952:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830381.239969:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973302400/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830381.239972:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973302784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830381.239993:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830381.239994:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830381.240000:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830381.240002:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787830381.240003:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830381.240009:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973302400:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787830381.240010:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6831 00000100:00080000:0.0:1787830381.240019:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830381.240025:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830381.240030:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830381.240038:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973302528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830381.240043:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973302912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830381.240053:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830381.240060:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830381.240062:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830381.240066:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6831 00000100:00080000:0.0:1787830381.240070:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830381.240075:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830381.240080:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787830381.240085:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830381.240096:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973303040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830381.240103:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830381/real 1787830381] req@ffff9c1a788621c0 x1874668973302528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830381.240109:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830381.240117:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830381.240121:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830381.240126:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6831 00000100:00100000:2.0:1787830381.240129:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973302528/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830381.240132:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830381.240140:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:2.0:1787830381.240146:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787830381.240148:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787830381.240153:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830381.240158:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973302528:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830381.240162:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973303168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830381.240168:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973302656:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830381.240195:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830381.240219:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830381/real 1787830381] req@ffff9c1a78862940 x1874668973302656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830381.240232:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973302656/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830381.240239:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830381.240242:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830381.240245:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973302656:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830381.240251:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973302784:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830381.240267:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830381.240277:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830381/real 1787830381] req@ffff9c1a78860780 x1874668973302784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830381.240287:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973302784/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830381.240295:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830381.240298:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830381.240301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973302784:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830381.240316:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973302912:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830381.240331:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830381.240340:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830381/real 1787830381] req@ffff9c1a78862d00 x1874668973302912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830381.240383:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973302912/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830381.240390:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830381.240393:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830381.240396:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973302912:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830381.240402:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973303040:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830381.240418:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830381.240427:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830381/real 1787830381] req@ffff9c1a78860b40 x1874668973303040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830381.240438:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973303040/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830381.240456:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830381.240459:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830381.240462:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973303040:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830381.240468:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973303168:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830381.240487:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830381.240496:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830381/real 1787830381] req@ffff9c1a78861a40 x1874668973303168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830381.240506:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973303168/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830381.240513:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830381.240516:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830381.240519:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973303168:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830381.751693:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830381.879560:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830382.071554:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830382.775486:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830382.903503:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830383.096306:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830383.799460:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830383.927439:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830384.119759:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830384.823476:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830384.952370:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830385.144225:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830385.847482:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830385.975553:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830386.168060:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830386.359493:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830386.359503:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830386.359507:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830386.359516:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6836 00000100:00080000:0.0:1787830386.359526:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830386.359533:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830386.359578:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973303296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830386.359604:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830386.359607:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830386.359609:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830386.359614:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6836 00000100:00080000:0.0:1787830386.359620:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830386.359623:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830386.359631:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830386.359640:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973303424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830386.359651:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830386.359653:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830386.359654:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830386.359656:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6836 00000100:00080000:0.0:1787830386.359658:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830386.359660:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830386.359662:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830386.359668:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973303552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830386.359674:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830386.359676:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830386.359677:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830386.359679:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6836 00000100:00080000:0.0:1787830386.359680:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830386.359682:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830386.359684:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830386.359691:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973303680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830386.359698:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830386.359700:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830386.359702:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830386.359705:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6836 00000100:00080000:0.0:1787830386.359708:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830386.359712:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830386.359716:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830386.359724:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973303808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830386.359732:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830386.359735:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830386.359736:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830386.359739:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6836 00000100:00080000:0.0:1787830386.359742:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830386.359745:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830386.359749:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830386.359759:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973303936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830386.359766:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830386.359769:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830386.359771:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830386.359774:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6836 00000100:00080000:0.0:1787830386.359777:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830386.359781:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830386.359797:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830386.359807:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973304064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830386.360736:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973303296:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830386.360795:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830386.360833:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830386/real 1787830386] req@ffff9c1a78863840 x1874668973303296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830386.360849:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973303296/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830386.360869:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830386.360876:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830386.360880:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973303296:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830386.360888:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973303424:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830386.360909:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830386.360921:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830386/real 1787830386] req@ffff9c1a78860000 x1874668973303424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830386.360933:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973303424/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830386.360941:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830386.360944:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830386.360947:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973303424:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830386.360953:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973303552:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830386.360970:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830386.360979:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830386/real 1787830386] req@ffff9c1a788630c0 x1874668973303552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830386.360990:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973303552/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830386.360997:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830386.361000:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830386.361003:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973303552:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830386.361009:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973303680:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830386.361027:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830386.361037:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830386/real 1787830386] req@ffff9c1a78862580 x1874668973303680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830386.361048:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973303680/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830386.361056:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830386.361059:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830386.361062:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973303680:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830386.361068:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973303808:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830386.361084:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830386.361093:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830386/real 1787830386] req@ffff9c1a78861680 x1874668973303808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830386.361103:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973303808/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830386.361111:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830386.361113:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830386.361116:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973303808:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830386.361122:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973303936:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830386.361141:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830386.361161:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830386/real 1787830386] req@ffff9c1a78861e00 x1874668973303936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830386.361186:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973303936/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830386.361194:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830386.361197:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830386.361200:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973303936:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830386.361205:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973304064:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830386.361225:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830386.361236:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830386/real 1787830386] req@ffff9c1a788612c0 x1874668973304064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830386.361248:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973304064/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830386.361256:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830386.361259:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830386.361262:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973304064:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830386.871497:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830386.999620:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830387.191773:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830387.895480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830388.023564:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830388.215619:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830388.919450:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830389.047438:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830389.239992:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830389.944314:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830390.071457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830390.264588:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830390.967738:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830391.095516:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830391.287605:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830391.479496:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830391.479506:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830391.479510:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830391.479521:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6841 00000100:00080000:0.0:1787830391.479533:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830391.479540:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830391.479575:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973304192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830391.479601:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830391.479604:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830391.479606:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830391.479610:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6841 00000100:00080000:0.0:1787830391.479614:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830391.479618:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830391.479627:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830391.479642:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973304320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830391.479652:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830391.479656:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830391.479658:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830391.479661:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6841 00000100:00080000:0.0:1787830391.479665:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830391.479669:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830391.479674:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830391.479685:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973304448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830391.479698:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830391.479699:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973304192:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787830391.479703:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830391.479708:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830391.479713:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6841 00000100:00080000:0.0:1787830391.479717:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830391.479721:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830391.479726:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830391.479739:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973304576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830391.479747:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830391.479751:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830391.479753:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787830391.479763:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830391.479771:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6841 00000100:00080000:0.0:1787830391.479777:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830391.479781:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830391.479787:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830391.479803:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973304704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830391.479812:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830391.479815:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830391/real 1787830391] req@ffff9c1a78863c00 x1874668973304192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830391.479825:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830391.479830:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830391.479836:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6841 00000100:00080000:0.0:1787830391.479841:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830391.479845:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973304192/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830391.479848:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830391.479857:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830391.479873:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973304832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830391.479881:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830391.479883:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830391.479889:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830391.479889:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:0.0:1787830391.479892:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830391.479897:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973304192:192.168.203.148@tcp:250:kworker.0 00000100:00080000:0.0:1787830391.479899:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6841 00000100:00080000:0.0:1787830391.479908:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830391.479914:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973304320:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830391.479921:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830391.479928:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830391.479942:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973304960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830391.479953:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830391.479976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830391/real 1787830391] req@ffff9c1a788612c0 x1874668973304320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830391.479993:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973304320/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830391.480003:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830391.480007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830391.480011:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973304320:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830391.480042:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973304448:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830391.480071:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830391.480084:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830391/real 1787830391] req@ffff9c1a78861e00 x1874668973304448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830391.480096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973304448/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830391.480103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830391.480107:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830391.480110:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973304448:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830391.480117:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973304576:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830391.480148:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830391.480159:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830391/real 1787830391] req@ffff9c1a78861680 x1874668973304576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830391.480169:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973304576/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830391.480177:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830391.480180:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830391.480183:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973304576:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830391.480187:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973304704:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830391.480204:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830391.480214:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830391/real 1787830391] req@ffff9c1a78862580 x1874668973304704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830391.480225:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973304704/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830391.480232:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830391.480235:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830391.480238:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973304704:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830391.480244:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973304832:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830391.480264:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830391.480274:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830391/real 1787830391] req@ffff9c1a788630c0 x1874668973304832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830391.480286:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973304832/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830391.480293:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830391.480297:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830391.480301:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973304832:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830391.480307:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973304960:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830391.480328:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830391.480340:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830391/real 1787830391] req@ffff9c1a78860000 x1874668973304960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830391.480387:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973304960/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830391.480395:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830391.480399:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830391.480402:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973304960:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830391.991522:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830392.119481:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830392.311873:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830393.015467:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830393.143478:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830393.335779:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830394.039495:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830394.167542:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830394.359596:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830395.063463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830395.191530:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830395.383969:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830396.087789:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830396.215483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830396.407656:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830396.599514:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830396.599525:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830396.599530:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830396.599540:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6846 00000100:00080000:0.0:1787830396.599550:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830396.599558:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830396.599593:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973305088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830396.599655:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973305088:192.168.203.148@tcp:250:kworker.0 00000800:00000100:2.0:1787830396.599713:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:0.0:1787830396.599746:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830396.599750:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830396.599753:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830396.599757:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6846 00000100:00080000:0.0:1787830396.599761:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830396.599766:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830396.599775:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830396.599789:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973305216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830396.599884:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830396/real 1787830396] req@ffff9c1a78863840 x1874668973305088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830396.599900:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973305088/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830396.599914:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830396.599917:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830396.599921:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973305088:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830396.599943:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973305216:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830396.600000:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830396.600015:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830396/real 1787830396] req@ffff9c1a78861a40 x1874668973305216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830396.600027:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973305216/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830396.600035:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830396.600038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830396.600041:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973305216:192.168.203.148@tcp:38:kworker.0 00000100:00080000:0.0:1787830396.600110:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830396.600114:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830396.600116:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830396.600120:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6846 00000100:00080000:0.0:1787830396.600125:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830396.600128:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830396.600133:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830396.600145:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973305344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830396.600157:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830396.600161:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830396.600163:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830396.600175:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6846 00000100:00080000:0.0:1787830396.600180:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830396.600183:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830396.600188:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830396.600198:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973305472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830396.600206:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830396.600209:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830396.600210:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830396.600213:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6846 00000100:00080000:0.0:1787830396.600216:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830396.600219:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830396.600223:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830396.600231:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973305600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830396.600244:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830396.600247:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830396.600248:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830396.600252:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6846 00000100:00080000:0.0:1787830396.600255:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830396.600259:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830396.600263:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830396.600273:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973305728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830396.600281:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830396.600284:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830396.600286:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830396.600289:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6846 00000100:00080000:0.0:1787830396.600292:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830396.600296:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830396.600300:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830396.600309:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973305856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830396.600646:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973305344:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830396.600671:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830396.600684:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830396/real 1787830396] req@ffff9c1a78860b40 x1874668973305344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830396.600695:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973305344/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830396.600703:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830396.600706:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830396.600709:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973305344:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830396.600715:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973305472:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830396.600733:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830396.600744:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830396/real 1787830396] req@ffff9c1a78862d00 x1874668973305472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830396.600755:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973305472/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830396.600762:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830396.600765:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830396.600768:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973305472:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830396.600774:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973305600:192.168.203.148@tcp:38:kworker.0 00000800:00000100:2.0:1787830396.600791:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830396.600801:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830396/real 1787830396] req@ffff9c1a78860780 x1874668973305600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830396.600812:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973305600/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830396.600820:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830396.600823:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830396.600826:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973305600:192.168.203.148@tcp:38:kworker.0 00000100:00100000:2.0:1787830396.600832:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973305728:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830396.600849:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830396.600860:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830396/real 1787830396] req@ffff9c1a78862940 x1874668973305728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830396.600872:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973305728/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830396.600879:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830396.600883:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830396.600886:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973305728:192.168.203.148@tcp:8:kworker.0 00000100:00100000:2.0:1787830396.600891:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973305856:192.168.203.148@tcp:8:kworker.0 00000800:00000100:2.0:1787830396.600907:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830396.600917:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830396/real 1787830396] req@ffff9c1a788621c0 x1874668973305856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830396.600928:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973305856/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830396.600936:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830396.600939:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:2.0:1787830396.600942:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973305856:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830397.112411:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830397.239465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830397.431669:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830398.135475:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830398.263843:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830398.455675:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830399.159463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830399.287745:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830399.480952:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830400.183432:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830400.311499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830400.503495:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830401.208083:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830401.335455:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830401.527558:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830401.722246:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830401.722256:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830401.722264:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830401.722277:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6852 00000100:00080000:2.0:1787830401.722288:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830401.722294:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830401.722329:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8de00 x1874668973305984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830401.722372:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830401.722376:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830401.722377:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830401.722381:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6852 00000100:00080000:2.0:1787830401.722385:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830401.722389:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830401.722398:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830401.722408:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973305984:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830401.722410:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c780 x1874668973306112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830401.722423:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830401.722426:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830401.722428:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830401.722431:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6852 00000100:00080000:2.0:1787830401.722434:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830401.722437:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830401.722441:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830401.722450:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cb40 x1874668973306240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830401.722459:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830401.722462:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830401.722464:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830401.722467:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6852 00000100:00080000:2.0:1787830401.722471:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830401.722475:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830401.722479:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787830401.722484:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:2.0:1787830401.722491:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8da40 x1874668973306368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830401.722500:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830401.722504:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830401.722505:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830401.722509:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6852 00000100:00080000:2.0:1787830401.722512:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830401.722514:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830401.722518:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830401.722527:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8cf00 x1874668973306496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830401.722534:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830401/real 1787830401] req@ffff9c1a49d8de00 x1874668973305984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830401.722536:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830401.722543:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830401.722546:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830401.722553:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6852 00000100:00080000:2.0:1787830401.722557:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830401.722560:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830401.722564:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830401.722574:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8de00 x1874668973305984/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830401.722575:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c000 x1874668973306624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830401.722604:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830401.722612:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830401.722614:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:2.0:1787830401.722618:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830401.722621:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830401.722622:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973305984:192.168.203.148@tcp:250:kworker.0 00000100:00080000:2.0:1787830401.722626:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6852 00000100:00080000:2.0:1787830401.722632:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830401.722637:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830401.722641:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830401.722650:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973306752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830401.722658:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973306112:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830401.722696:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830401.722710:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830401/real 1787830401] req@ffff9c1a49d8c780 x1874668973306112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830401.722721:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c780 x1874668973306112/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830401.722729:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830401.722732:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830401.722735:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973306112:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830401.722740:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973306240:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830401.722759:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830401.722781:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830401/real 1787830401] req@ffff9c1a49d8cb40 x1874668973306240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830401.722792:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cb40 x1874668973306240/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830401.722800:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830401.722803:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830401.722806:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973306240:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830401.722812:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973306368:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830401.722831:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830401.722841:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830401/real 1787830401] req@ffff9c1a49d8da40 x1874668973306368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830401.722852:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8da40 x1874668973306368/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830401.722860:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830401.722863:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830401.722866:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973306368:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830401.722871:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973306496:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830401.722890:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830401.722900:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830401/real 1787830401] req@ffff9c1a49d8cf00 x1874668973306496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830401.722910:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8cf00 x1874668973306496/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830401.722915:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830401.722917:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830401.722918:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973306496:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830401.722922:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973306624:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830401.722936:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830401.722945:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830401/real 1787830401] req@ffff9c1a49d8c000 x1874668973306624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830401.722955:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c000 x1874668973306624/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830401.722967:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830401.722970:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830401.722973:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973306624:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830401.722997:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973306752:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830401.723015:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830401.723026:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830401/real 1787830401] req@ffff9c1a49d8e1c0 x1874668973306752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830401.723037:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973306752/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830401.723053:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830401.723064:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830401.723068:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973306752:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830402.231606:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830402.359694:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830402.551525:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830403.255454:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830403.383603:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830403.576264:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830404.279518:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830404.407502:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830404.600715:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830405.303494:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830405.431477:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830405.623826:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830406.327575:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830406.456061:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830406.647638:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830406.840076:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830406.840088:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830406.840093:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830406.840103:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6857 00000100:00080000:0.0:1787830406.840115:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830406.840121:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830406.840167:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860f00 x1874668973306880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830406.840199:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830406.840202:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830406.840204:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830406.840208:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6857 00000100:00080000:0.0:1787830406.840212:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830406.840215:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830406.840225:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830406.840238:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973307008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830406.840252:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830406.840255:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830406.840256:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830406.840259:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6857 00000100:00080000:0.0:1787830406.840263:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830406.840266:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830406.840270:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830406.840278:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973307136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830406.840287:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830406.840290:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830406.840292:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830406.840294:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6857 00000100:00080000:0.0:1787830406.840297:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830406.840300:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830406.840304:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830406.840313:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973307264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830406.840321:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830406.840324:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830406.840325:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830406.840328:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6857 00000100:00080000:0.0:1787830406.840332:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830406.840335:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830406.840340:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830406.840378:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973307392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830406.840387:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830406.840389:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830406.840391:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830406.840399:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6857 00000100:00080000:0.0:1787830406.840403:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830406.840407:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830406.840411:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830406.840421:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973307520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830406.840429:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830406.840432:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830406.840433:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830406.840437:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6857 00000100:00080000:0.0:1787830406.840440:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830406.840443:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830406.840447:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830406.840456:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973307648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830406.840546:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973306880:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830406.840598:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830406.840631:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830406/real 1787830406] req@ffff9c1a78860f00 x1874668973306880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830406.840645:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860f00 x1874668973306880/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830406.840659:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830406.840662:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830406.840666:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973306880:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830406.840672:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973307008:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830406.840692:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830406.840704:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830406/real 1787830406] req@ffff9c1a788621c0 x1874668973307008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830406.840715:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973307008/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830406.840722:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830406.840725:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830406.840728:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973307008:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830406.840734:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973307136:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830406.840750:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830406.840760:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830406/real 1787830406] req@ffff9c1a78862940 x1874668973307136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830406.840769:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973307136/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830406.840776:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830406.840779:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830406.840782:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973307136:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830406.840788:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973307264:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830406.840805:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830406.840815:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830406/real 1787830406] req@ffff9c1a78860780 x1874668973307264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830406.840825:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973307264/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830406.840832:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830406.840834:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830406.840838:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973307264:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830406.840843:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973307392:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830406.840859:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830406.840868:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830406/real 1787830406] req@ffff9c1a78862d00 x1874668973307392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830406.840878:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973307392/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830406.840885:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830406.840888:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830406.840890:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973307392:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830406.840895:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973307520:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830406.840912:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830406.840921:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830406/real 1787830406] req@ffff9c1a78860b40 x1874668973307520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830406.840932:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973307520/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830406.840938:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830406.840941:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830406.840944:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973307520:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830406.840949:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973307648:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830406.840968:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830406.840978:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830406/real 1787830406] req@ffff9c1a78861a40 x1874668973307648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830406.840988:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973307648/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830406.840995:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830406.840998:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830406.841001:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973307648:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830407.351582:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830407.479575:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830407.673538:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830408.375481:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830408.503522:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830408.695575:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830409.399674:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830409.527466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830409.719905:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830410.423518:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830410.551578:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830410.745090:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830411.447463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830411.575514:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830411.767613:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830411.959555:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830411.959564:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830411.959568:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830411.960148:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6862 00000100:00080000:0.0:1787830411.960159:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830411.960165:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830411.960207:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973307776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830411.960228:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830411.960230:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830411.960232:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830411.960235:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6862 00000100:00080000:0.0:1787830411.960237:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830411.960239:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830411.960246:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830411.960254:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973307904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830411.960260:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830411.960262:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830411.960263:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830411.960265:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6862 00000100:00080000:0.0:1787830411.960267:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830411.960269:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830411.960271:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830411.960276:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973308032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830411.960282:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830411.960284:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830411.960285:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830411.960287:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6862 00000100:00080000:0.0:1787830411.960289:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830411.960291:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830411.960293:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830411.960298:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973308160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830411.960302:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830411.960304:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830411.960305:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830411.960306:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6862 00000100:00080000:0.0:1787830411.960308:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830411.960310:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830411.960311:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830411.960317:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973308288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830411.960322:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830411.960325:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830411.960327:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830411.960330:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6862 00000100:00080000:0.0:1787830411.960333:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830411.960337:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830411.960341:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830411.960380:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973308416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830411.960390:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830411.960393:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830411.960394:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830411.960398:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6862 00000100:00080000:0.0:1787830411.960401:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830411.960404:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830411.960409:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830411.960419:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973308544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830411.960663:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973307776:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830411.960750:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830411.960800:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830411/real 1787830411] req@ffff9c1a78863840 x1874668973307776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830411.960820:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973307776/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830411.960832:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830411.960837:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830411.960841:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973307776:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830411.960849:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973307904:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830411.960873:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830411.960885:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830411/real 1787830411] req@ffff9c1a78860000 x1874668973307904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830411.960897:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973307904/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830411.960906:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830411.960910:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830411.960913:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973307904:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830411.960919:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973308032:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830411.960936:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830411.960946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830411/real 1787830411] req@ffff9c1a788630c0 x1874668973308032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830411.960964:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973308032/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830411.960971:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830411.960975:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830411.960978:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973308032:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830411.960992:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973308160:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830411.961014:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830411.961024:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830411/real 1787830411] req@ffff9c1a78862580 x1874668973308160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830411.961035:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973308160/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830411.961043:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830411.961046:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830411.961049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973308160:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830411.961055:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973308288:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830411.961074:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830411.961085:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830411/real 1787830411] req@ffff9c1a78861680 x1874668973308288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830411.961096:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973308288/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830411.961110:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830411.961113:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830411.961116:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973308288:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830411.961123:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973308416:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830411.961142:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830411.961152:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830411/real 1787830411] req@ffff9c1a78861e00 x1874668973308416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830411.961163:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973308416/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830411.961171:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830411.961174:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830411.961177:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973308416:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830411.961183:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973308544:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830411.961240:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830411.961251:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830411/real 1787830411] req@ffff9c1a788612c0 x1874668973308544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830411.961263:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973308544/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830411.961271:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830411.961274:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830411.961278:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973308544:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830412.471471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830412.600855:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830412.791688:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830413.496826:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830413.623511:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830413.815873:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830414.519471:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830414.647510:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830414.840217:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830415.543958:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830415.671923:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830415.863774:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830416.567513:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830416.696701:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830416.887682:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830417.079485:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830417.079495:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830417.079500:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830417.079510:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6867 00000100:00080000:0.0:1787830417.079522:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830417.079531:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830417.079582:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863c00 x1874668973308672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830417.079613:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830417.079616:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830417.079618:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830417.079622:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6867 00000100:00080000:0.0:1787830417.079626:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830417.079630:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830417.079639:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830417.079652:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788612c0 x1874668973308800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830417.079672:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830417.079675:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830417.079676:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830417.079683:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6867 00000100:00080000:0.0:1787830417.079687:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830417.079691:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830417.079695:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830417.079705:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861e00 x1874668973308928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830417.079715:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830417.079718:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830417.079720:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830417.079723:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6867 00000100:00080000:0.0:1787830417.079726:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830417.079730:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830417.079734:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830417.079744:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861680 x1874668973309056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830417.079752:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830417.079755:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830417.079757:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830417.079760:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6867 00000100:00080000:0.0:1787830417.079763:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830417.079766:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830417.079770:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830417.079779:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862580 x1874668973309184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830417.079788:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830417.079791:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830417.079792:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830417.079795:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6867 00000100:00080000:0.0:1787830417.079799:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830417.079802:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830417.079807:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830417.079816:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788630c0 x1874668973309312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830417.079824:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830417.079827:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830417.079828:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830417.079832:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6867 00000100:00080000:0.0:1787830417.079835:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830417.079838:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830417.079842:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830417.079851:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860000 x1874668973309440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830417.079901:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973308672:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830417.079966:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830417.080004:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830417/real 1787830417] req@ffff9c1a78863c00 x1874668973308672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830417.080020:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863c00 x1874668973308672/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830417.080034:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830417.080038:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830417.080042:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973308672:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830417.080048:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973308800:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830417.080070:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830417.080081:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830417/real 1787830417] req@ffff9c1a788612c0 x1874668973308800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830417.080093:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788612c0 x1874668973308800/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830417.080105:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830417.080107:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830417.080109:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788612c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973308800:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830417.080113:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973308928:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830417.080126:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830417.080133:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830417/real 1787830417] req@ffff9c1a78861e00 x1874668973308928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830417.080139:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861e00 x1874668973308928/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830417.080144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830417.080146:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830417.080148:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973308928:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830417.080151:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973309056:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830417.080160:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830417.080165:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830417/real 1787830417] req@ffff9c1a78861680 x1874668973309056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830417.080172:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861680 x1874668973309056/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830417.080176:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830417.080178:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830417.080180:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973309056:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830417.080183:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973309184:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830417.080191:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830417.080196:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830417/real 1787830417] req@ffff9c1a78862580 x1874668973309184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830417.080202:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862580 x1874668973309184/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830417.080206:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830417.080207:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830417.080209:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973309184:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830417.080212:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973309312:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830417.080220:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830417.080236:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830417/real 1787830417] req@ffff9c1a788630c0 x1874668973309312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830417.080243:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788630c0 x1874668973309312/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830417.080248:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830417.080250:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830417.080252:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788630c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973309312:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830417.080255:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973309440:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830417.080265:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830417.080271:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830417/real 1787830417] req@ffff9c1a78860000 x1874668973309440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830417.080277:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860000 x1874668973309440/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830417.080281:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830417.080283:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830417.080284:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973309440:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830417.592054:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830417.719501:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830417.911552:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830418.615473:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830418.744020:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830418.936487:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830419.639466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830419.767457:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830419.959981:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830420.663431:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830420.791472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830420.984125:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830421.688257:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830421.815465:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830422.008291:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830422.199538:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830422.199549:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830422.199553:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830422.199562:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6872 00000100:00080000:1.0:1787830422.199573:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830422.199579:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830422.199613:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973309568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830422.199842:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973309568:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830422.199897:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830422.199934:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830422.199939:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830422.199941:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830422.199945:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6872 00000100:00080000:1.0:1787830422.199949:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830422.199953:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830422.199961:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830422.199974:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973309696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830422.200078:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830422/real 1787830422] req@ffff9c1a78a7a580 x1874668973309568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830422.200095:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973309568/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830422.200103:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830422.200106:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830422.200109:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973309568:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830422.200130:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973309696:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830422.200151:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830422.200162:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830422/real 1787830422] req@ffff9c1a78a78f00 x1874668973309696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830422.200173:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973309696/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830422.200180:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830422.200183:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830422.200186:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973309696:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830422.200318:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830422.200322:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830422.200323:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830422.200327:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6872 00000100:00080000:1.0:1787830422.200332:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830422.200335:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830422.200340:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830422.200377:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973309824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830422.200424:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973309824:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830422.200445:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830422.200466:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830422.200470:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830422.200471:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830422.200475:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6872 00000100:00080000:1.0:1787830422.200479:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830422.200483:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830422.200488:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830422.200497:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973309952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830422.200532:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830422/real 1787830422] req@ffff9c1a78a7b480 x1874668973309824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830422.200539:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973309824/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830422.200544:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830422.200546:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830422.200548:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973309824:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830422.200556:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973309952:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830422.200569:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830422.200575:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830422/real 1787830422] req@ffff9c1a78a79a40 x1874668973309952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830422.200585:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973309952/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830422.200590:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830422.200592:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830422.200594:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973309952:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830422.200773:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830422.200776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830422.200777:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830422.200780:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6872 00000100:00080000:1.0:1787830422.200783:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830422.200786:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830422.200790:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830422.200798:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973310080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830422.200869:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973310080:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830422.200890:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830422.200908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830422.200911:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830422.200912:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830422.200925:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6872 00000100:00080000:1.0:1787830422.200928:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830422.200932:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830422.200937:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830422.200949:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973310208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830422.200986:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830422/real 1787830422] req@ffff9c1a78a78000 x1874668973310080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830422.200997:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973310080/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830422.201004:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830422.201007:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830422.201010:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973310080:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830422.201022:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973310208:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830422.201041:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830422.201052:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830422/real 1787830422] req@ffff9c1a78a792c0 x1874668973310208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830422.201126:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973310208/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830422.201141:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830422.201144:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830422.201147:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973310208:192.168.203.148@tcp:8:kworker.0 00000100:00080000:1.0:1787830422.201321:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830422.201325:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830422.201327:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830422.201331:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6872 00000100:00080000:1.0:1787830422.201335:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830422.201339:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830422.201363:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830422.201376:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973310336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830422.201435:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973310336:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830422.201458:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830422.201469:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830422/real 1787830422] req@ffff9c1a78a7b840 x1874668973310336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830422.201480:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973310336/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830422.201487:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830422.201489:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830422.201492:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973310336:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830422.711542:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830422.839756:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830423.032118:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830423.735480:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830423.863456:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830424.055655:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830424.759506:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830424.887486:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830425.080139:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830425.783469:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830425.911466:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830426.103710:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830426.807448:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830426.935553:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830427.129126:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830427.321125:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830427.321134:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830427.321139:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830427.321158:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6877 00000100:00080000:0.0:1787830427.321168:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830427.321176:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830427.321209:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78863840 x1874668973310464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830427.321239:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830427.321243:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830427.321245:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830427.321249:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6877 00000100:00080000:0.0:1787830427.321253:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830427.321256:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830427.321265:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830427.321277:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78861a40 x1874668973310592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830427.321287:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830427.321290:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830427.321292:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830427.321295:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6877 00000100:00080000:0.0:1787830427.321298:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830427.321301:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830427.321305:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830427.321315:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860b40 x1874668973310720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830427.321325:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830427.321328:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830427.321329:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830427.321332:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6877 00000100:00080000:0.0:1787830427.321335:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830427.321339:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830427.321360:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830427.321372:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862d00 x1874668973310848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830427.321380:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830427.321383:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830427.321385:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830427.321388:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6877 00000100:00080000:0.0:1787830427.321391:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830427.321394:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830427.321398:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830427.321406:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78860780 x1874668973310976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830427.321414:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830427.321417:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830427.321418:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830427.321422:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6877 00000100:00080000:0.0:1787830427.321425:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830427.321428:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830427.321433:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830427.321441:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78862940 x1874668973311104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830427.321447:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830427.321449:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830427.321450:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830427.321452:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6877 00000100:00080000:0.0:1787830427.321454:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830427.321456:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:0.0:1787830427.321459:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830427.321463:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a788621c0 x1874668973311232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830427.321676:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973310464:192.168.203.148@tcp:250:kworker.0 00000800:00000100:3.0:1787830427.321782:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830427.321806:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830427/real 1787830427] req@ffff9c1a78863840 x1874668973310464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830427.321827:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78863840 x1874668973310464/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830427.321836:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830427.321840:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830427.321844:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78863840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973310464:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830427.321855:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973310592:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830427.321874:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830427.321897:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830427/real 1787830427] req@ffff9c1a78861a40 x1874668973310592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830427.321908:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78861a40 x1874668973310592/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830427.321913:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830427.321915:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830427.321917:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78861a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973310592:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830427.321921:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973310720:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830427.321935:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830427.321941:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830427/real 1787830427] req@ffff9c1a78860b40 x1874668973310720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830427.321948:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860b40 x1874668973310720/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830427.321956:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830427.321958:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830427.321960:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973310720:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830427.321962:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973310848:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830427.321971:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830427.321976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830427/real 1787830427] req@ffff9c1a78862d00 x1874668973310848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830427.321982:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862d00 x1874668973310848/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830427.321986:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830427.321988:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830427.321989:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973310848:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830427.321992:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973310976:192.168.203.148@tcp:38:kworker.0 00000800:00000100:3.0:1787830427.322001:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830427.322006:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830427/real 1787830427] req@ffff9c1a78860780 x1874668973310976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830427.322011:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78860780 x1874668973310976/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830427.322016:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830427.322017:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830427.322020:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78860780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973310976:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830427.322023:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973311104:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830427.322032:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830427.322036:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830427/real 1787830427] req@ffff9c1a78862940 x1874668973311104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830427.322042:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78862940 x1874668973311104/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830427.322047:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830427.322048:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830427.322049:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78862940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973311104:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830427.322052:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973311232:192.168.203.148@tcp:8:kworker.0 00000800:00000100:3.0:1787830427.322060:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:3.0:1787830427.322064:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830427/real 1787830427] req@ffff9c1a788621c0 x1874668973311232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830427.322071:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a788621c0 x1874668973311232/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830427.322075:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830427.322076:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830427.322078:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a788621c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973311232:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830427.831453:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830427.959950:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830428.151632:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830428.855684:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830428.983472:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830429.175640:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830429.879987:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830430.007462:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830430.199656:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830430.903764:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830431.031492:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830431.223616:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830431.927963:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830432.056969:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830432.248226:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830432.439653:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830432.439662:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830432.439666:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830432.439675:0:40547:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6882 00000100:00080000:2.0:1787830432.439685:0:40547:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830432.439693:0:40547:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:2.0:1787830432.439727:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8f0c0 x1874668973311360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830432.439760:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830432.439764:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830432.439766:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830432.439770:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6882 00000100:00080000:2.0:1787830432.439774:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830432.439778:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830432.439787:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830432.439799:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8c3c0 x1874668973311488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830432.439808:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830432.439812:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830432.439814:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830432.439817:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6882 00000100:00080000:2.0:1787830432.439820:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830432.439823:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830432.439828:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830432.439837:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8d680 x1874668973311616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830432.439858:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830432.439862:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830432.439863:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830432.439867:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6882 00000100:00080000:2.0:1787830432.439870:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830432.439874:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830432.439878:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830432.439885:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973311360:192.168.203.148@tcp:250:kworker.0 00000100:00100000:2.0:1787830432.439887:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8fc00 x1874668973311744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830432.439895:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830432.439898:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830432.439899:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830432.439902:0:40547:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6882 00000100:00080000:2.0:1787830432.439904:0:40547:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830432.439907:0:40547:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830432.439911:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830432.439918:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8ed00 x1874668973311872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830432.439925:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830432.439927:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830432.439929:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830432.439932:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6882 00000100:00080000:2.0:1787830432.439934:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830432.439938:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830432.439942:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830432.439951:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e580 x1874668973312000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830432.439959:0:40547:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830432.439961:0:40547:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830432.439963:0:40547:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830432.439966:0:40547:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6882 00000100:00080000:2.0:1787830432.439970:0:40547:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830432.439973:0:40547:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:2.0:1787830432.439977:0:40547:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830432.439987:0:40547:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a49d8e1c0 x1874668973312128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830432.440732:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973311488:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830432.440769:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973311616:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830432.440800:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973311744:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830432.440819:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973311872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830432.440838:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973312000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830432.440872:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973312128:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830432.951623:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830433.079578:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830433.271608:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830433.975819:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830434.103523:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830434.295586:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830434.999916:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830435.127411:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830435.319595:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787830435.512263:0:2401:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.148:988 00000400:00000100:1.0:1787830435.512389:0:2401:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.148@tcp at host 192.168.203.148 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830435.512412:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787830435.512562:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830432/real 1787830435] req@ffff9c1a49d8f0c0 x1874668973311360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830457 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830435.512596:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8f0c0 x1874668973311360/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830457 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830435.512609:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830435.512614:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830435.512619:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973311360:192.168.203.148@tcp:250:kworker.0 00000800:00000100:1.0:1787830435.512706:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787830435.512770:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830432/real 1787830435] req@ffff9c1a49d8c3c0 x1874668973311488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830457 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830435.512784:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8c3c0 x1874668973311488/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830457 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830435.512791:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830435.512796:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830435.512799:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973311488:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830435.512851:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787830435.512960:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830432/real 1787830435] req@ffff9c1a49d8d680 x1874668973311616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830457 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830435.512979:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8d680 x1874668973311616/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830457 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830435.512987:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830435.512991:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830435.512995:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973311616:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830435.513129:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787830435.515069:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830432/real 1787830435] req@ffff9c1a49d8fc00 x1874668973311744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830457 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830435.515115:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8fc00 x1874668973311744/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830457 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830435.515128:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830435.515134:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830435.515138:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973311744:192.168.203.148@tcp:38:kworker.0 00000800:00000100:1.0:1787830435.515373:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830435.515409:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000800:00000100:1.0:1787830435.515418:0:2401:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.148@tcp 00000100:00100000:3.0:1787830435.515541:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830432/real 1787830435] req@ffff9c1a49d8ed00 x1874668973311872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830435.515559:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8ed00 x1874668973311872/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830435.515567:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830435.515573:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830435.515576:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973311872:192.168.203.148@tcp:38:kworker.0 00000100:00100000:3.0:1787830435.515584:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830432/real 1787830435] req@ffff9c1a49d8e580 x1874668973312000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830435.515599:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e580 x1874668973312000/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830435.515608:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830435.515611:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830435.515614:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973312000:192.168.203.148@tcp:8:kworker.0 00000100:00100000:3.0:1787830435.515620:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830432/real 1787830435] req@ffff9c1a49d8e1c0 x1874668973312128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830435.515631:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a49d8e1c0 x1874668973312128/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830435.515638:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830435.515642:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:3.0:1787830435.515645:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a49d8e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973312128:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830436.023527:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830436.151499:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830436.343602:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830437.047407:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830437.176020:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830437.367691:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830437.559748:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830437.559762:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830437.559768:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830437.559782:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6887 00000100:00080000:3.0:1787830437.559797:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830437.559805:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830437.559870:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895d2c0 x1874668973312256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830437.559912:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830437.559916:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830437.559918:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830437.559922:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6887 00000100:00080000:3.0:1787830437.559926:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830437.559930:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830437.559941:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830437.559955:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973312384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830437.559965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830437.559969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830437.559970:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830437.559974:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6887 00000100:00080000:3.0:1787830437.559979:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830437.559984:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830437.559991:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830437.560005:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973312256:192.168.203.148@tcp:250:kworker.0 00000100:00100000:3.0:1787830437.560005:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973312512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830437.560044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830437.560049:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830437.560051:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830437.560056:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6887 00000100:00080000:3.0:1787830437.560062:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830437.560067:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830437.560073:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830437.560088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973312640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830437.560098:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830437.560102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830437.560106:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787830437.560108:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:3.0:1787830437.560118:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6887 00000100:00080000:3.0:1787830437.560124:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830437.560130:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830437.560136:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830437.560152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973312768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830437.560163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830437.560169:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787830437.560170:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830437/real 1787830437] req@ffff9c1a7895d2c0 x1874668973312256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830437.560173:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830437.560182:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6887 00000100:00080000:3.0:1787830437.560193:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830437.560204:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830437.560211:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830437.560216:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895d2c0 x1874668973312256/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830437.560229:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973312896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830437.560231:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830437.560246:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830437.560257:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973312256:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787830437.560274:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830437.560280:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973312384:192.168.203.148@tcp:38:kworker.0 00000100:00080000:3.0:1787830437.560282:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830437.560287:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830437.560302:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6887 00000100:00080000:3.0:1787830437.560310:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830437.560316:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000800:00000100:0.0:1787830437.560322:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000008:00100000:3.0:1787830437.560323:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830437.560399:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973313024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830437.560698:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830437/real 1787830437] req@ffff9c1a7895de00 x1874668973312384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830437.560722:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973312384/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830437.560732:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830437.560736:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830437.560740:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973312384:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830437.560771:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973312512:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830437.560800:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830437.560824:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830437/real 1787830437] req@ffff9c1a7895f840 x1874668973312512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830437.560837:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973312512/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830437.560849:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830437.560853:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830437.560856:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973312512:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830437.560862:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973312640:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830437.560881:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830437.560893:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830437/real 1787830437] req@ffff9c1a7895f0c0 x1874668973312640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830437.560903:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973312640/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830437.560910:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830437.560913:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830437.560916:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973312640:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830437.560923:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973312768:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830437.563930:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830437.563964:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830437/real 1787830437] req@ffff9c1a7895fc00 x1874668973312768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830437.563983:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973312768/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830437.563994:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830437.563999:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830437.564003:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973312768:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830437.564014:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973312896:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830437.564040:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830437.564051:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830437/real 1787830437] req@ffff9c1a7895cb40 x1874668973312896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830437.564062:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973312896/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830437.564070:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830437.564074:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830437.564077:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973312896:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830437.564083:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973313024:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830437.564103:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830437.564113:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830437/real 1787830437] req@ffff9c1a7895e1c0 x1874668973313024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830437.564124:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973313024/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830437.564131:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830437.564135:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830437.564138:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973313024:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830438.071474:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830438.199678:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830438.391865:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830439.095468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830439.223471:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830439.415613:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830440.119868:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830440.248036:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830440.439547:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830441.143780:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830441.272809:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830441.463558:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830442.167468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830442.295650:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830442.487563:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830442.679641:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830442.679652:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830442.679656:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830442.679666:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6893 00000100:00080000:1.0:1787830442.679740:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830442.679750:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830442.679803:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b840 x1874668973313152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830442.679882:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830442.679886:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830442.679888:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830442.679893:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6893 00000100:00080000:1.0:1787830442.679897:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830442.679901:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830442.679910:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830442.679924:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a792c0 x1874668973313280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830442.679934:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830442.679937:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830442.679939:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830442.679943:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6893 00000100:00080000:1.0:1787830442.679946:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830442.679949:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830442.679954:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830442.679964:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78000 x1874668973313408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830442.679974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830442.679978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830442.679979:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830442.679983:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6893 00000100:00080000:1.0:1787830442.679986:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830442.679989:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830442.679993:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830442.680003:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79a40 x1874668973313536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830442.680011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830442.680014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830442.680016:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830442.680019:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6893 00000100:00080000:1.0:1787830442.680023:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830442.680026:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830442.680031:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830442.680041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b480 x1874668973313664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830442.680049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830442.680052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830442.680054:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830442.680057:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6893 00000100:00080000:1.0:1787830442.680060:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830442.680064:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830442.680082:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830442.680092:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78f00 x1874668973313792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830442.680100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830442.680103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830442.680105:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830442.680107:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6893 00000100:00080000:1.0:1787830442.680121:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830442.680125:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830442.680130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830442.680140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a580 x1874668973313920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830442.680227:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973313152:192.168.203.148@tcp:250:kworker.0 00000800:00000100:0.0:1787830442.680294:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830442.680341:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830442/real 1787830442] req@ffff9c1a78a7b840 x1874668973313152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830442.680380:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b840 x1874668973313152/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830442.680389:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830442.680393:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830442.680396:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973313152:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830442.680403:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973313280:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830442.680425:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830442.680436:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830442/real 1787830442] req@ffff9c1a78a792c0 x1874668973313280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830442.680461:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a792c0 x1874668973313280/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830442.680469:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830442.680472:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830442.680475:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a792c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973313280:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830442.680484:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973313408:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830442.680504:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830442.680946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830442/real 1787830442] req@ffff9c1a78a78000 x1874668973313408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830442.680961:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78000 x1874668973313408/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830442.680968:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830442.680972:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830442.680975:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973313408:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830442.680981:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973313536:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830442.681002:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830442.681014:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830442/real 1787830442] req@ffff9c1a78a79a40 x1874668973313536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830442.681024:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79a40 x1874668973313536/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830442.681031:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830442.681034:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830442.681037:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973313536:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830442.681043:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973313664:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830442.681061:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830442.681182:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830442/real 1787830442] req@ffff9c1a78a7b480 x1874668973313664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830442.681193:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b480 x1874668973313664/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830442.681202:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830442.681205:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830442.681208:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973313664:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830442.681213:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973313792:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830442.681235:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830442.681305:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830442/real 1787830442] req@ffff9c1a78a78f00 x1874668973313792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830442.681487:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78f00 x1874668973313792/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830442.681495:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830442.681498:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830442.681501:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973313792:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830442.681507:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973313920:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830442.681530:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830442.681541:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830442/real 1787830442] req@ffff9c1a78a7a580 x1874668973313920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830442.681551:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a580 x1874668973313920/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830442.681565:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830442.681568:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830442.681571:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973313920:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830443.191895:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830443.319521:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830443.512303:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830444.215468:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830444.343483:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830444.536258:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830445.240014:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830445.367726:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830445.559708:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830446.263458:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830446.391775:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830446.583671:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830447.287472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830447.415458:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830447.607727:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830447.800966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830447.800976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830447.800981:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830447.800990:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6898 00000100:00080000:3.0:1787830447.801001:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830447.801007:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:3.0:1787830447.801041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895da40 x1874668973314048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830447.801064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830447.801077:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830447.801078:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830447.801082:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6898 00000100:00080000:3.0:1787830447.801086:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830447.801090:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830447.801099:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830447.801115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895e1c0 x1874668973314176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830447.801124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830447.801128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830447.801130:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830447.801133:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6898 00000100:00080000:3.0:1787830447.801136:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830447.801140:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830447.801144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830447.801155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895cb40 x1874668973314304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830447.801171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830447.801174:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830447.801176:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830447.801179:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6898 00000100:00080000:3.0:1787830447.801187:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830447.801190:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830447.801195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830447.801205:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895fc00 x1874668973314432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830447.801214:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830447.801217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830447.801218:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830447.801222:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6898 00000100:00080000:3.0:1787830447.801229:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830447.801232:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830447.801236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830447.801245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f0c0 x1874668973314560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830447.801254:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830447.801256:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830447.801257:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830447.801260:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6898 00000100:00080000:3.0:1787830447.801263:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830447.801264:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973314048:192.168.203.148@tcp:250:kworker.0 00000100:00080000:3.0:1787830447.801267:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830447.801271:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830447.801278:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895f840 x1874668973314688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830447.801286:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830447.801288:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830447.801290:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830447.801292:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6898 00000100:00080000:3.0:1787830447.801294:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830447.801297:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:3.0:1787830447.801300:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830447.801308:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a7895de00 x1874668973314816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830447.801318:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830447.801387:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830447/real 1787830447] req@ffff9c1a7895da40 x1874668973314048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830447.801403:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895da40 x1874668973314048/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830447.801412:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830447.801416:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830447.801419:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973314048:192.168.203.148@tcp:250:kworker.0 00000100:00100000:0.0:1787830447.801426:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973314176:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830447.801453:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830447.801465:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830447/real 1787830447] req@ffff9c1a7895e1c0 x1874668973314176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830447.801650:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895e1c0 x1874668973314176/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830447.801659:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830447.801663:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830447.801666:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973314176:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830447.801672:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973314304:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830447.801692:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830447.801703:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830447/real 1787830447] req@ffff9c1a7895cb40 x1874668973314304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830447.801712:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895cb40 x1874668973314304/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830447.801718:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830447.801721:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830447.801723:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973314304:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830447.801728:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973314432:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830447.801756:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830447.801765:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830447/real 1787830447] req@ffff9c1a7895fc00 x1874668973314432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830447.801774:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895fc00 x1874668973314432/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830447.801781:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830447.801784:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830447.801787:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973314432:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830447.801792:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973314560:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830447.801805:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830447.801812:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830447/real 1787830447] req@ffff9c1a7895f0c0 x1874668973314560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830447.801821:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f0c0 x1874668973314560/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830447.801827:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830447.801829:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830447.801832:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973314560:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830447.801854:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973314688:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830447.801870:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830447.801879:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830447/real 1787830447] req@ffff9c1a7895f840 x1874668973314688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830447.801889:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895f840 x1874668973314688/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830447.801895:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830447.801898:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830447.801901:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973314688:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830447.801906:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973314816:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830447.801921:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830447.801930:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830447/real 1787830447] req@ffff9c1a7895de00 x1874668973314816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830447.801940:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a7895de00 x1874668973314816/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830447.801948:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830447.801950:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830447.801953:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a7895de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973314816:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830448.312529:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830448.439646:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830448.631834:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830449.335556:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830449.463491:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830449.655614:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830450.359491:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830450.488419:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830450.679521:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830451.383472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830451.511583:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830451.704759:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830452.407493:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830452.535560:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830452.727751:0:6180:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830452.920327:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830452.920338:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830452.920342:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830452.920371:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6903 00000100:00080000:1.0:1787830452.920382:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830452.920389:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830452.920428:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973314944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830452.920455:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830452.920463:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830452.920465:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830452.920469:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6903 00000100:00080000:1.0:1787830452.920473:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830452.920477:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830452.920487:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830452.920500:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973314944:192.168.203.148@tcp:250:kworker.0 00000100:00100000:1.0:1787830452.920502:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973315072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830452.920515:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830452.920518:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830452.920520:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830452.920524:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6903 00000100:00080000:1.0:1787830452.920528:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830452.920532:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830452.920536:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830452.920547:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973315200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830452.920555:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830452.920563:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830452.920566:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830452.920569:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830452.920574:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6903 00000100:00080000:1.0:1787830452.920578:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830452.920582:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830452.920587:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830452.920593:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830452/real 1787830452] req@ffff9c1a78a7bc00 x1874668973314944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830452.920600:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973315328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830452.920611:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830452.920614:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830452.920616:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830452.920617:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973314944/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830452.920625:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6903 00000100:00080000:1.0:1787830452.920630:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830452.920633:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830452.920635:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00080000:0.0:1787830452.920638:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830452.920642:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973314944:192.168.203.148@tcp:250:kworker.0 00000008:00100000:1.0:1787830452.920643:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830452.920657:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973315456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830452.920664:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973315072:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830452.920670:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830452.920675:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830452.920677:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830452.920683:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6903 00000800:00000100:0.0:1787830452.920686:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830452.920690:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830452.920696:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:0.0:1787830452.920698:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830452/real 1787830452] req@ffff9c1a78a78780 x1874668973315072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787830452.920716:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830452.920725:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973315072/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830452.920732:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973315584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830452.920742:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830452.920746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830452.920749:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787830452.920753:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830452.920757:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830452.920760:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973315072:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830452.920763:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6903 00000100:00100000:0.0:1787830452.920768:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973315200:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830452.920769:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830452.920776:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830452.920782:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787830452.920790:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:1.0:1787830452.920797:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973315712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830452.920801:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830452/real 1787830452] req@ffff9c1a78a7a940 x1874668973315200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830452.920823:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973315200/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830452.920828:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830452.920830:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830452.920832:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973315200:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830452.920837:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973315328:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830452.920854:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830452.920863:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830452/real 1787830452] req@ffff9c1a78a79e00 x1874668973315328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830452.920875:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973315328/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830452.920887:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830452.920890:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830452.920893:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973315328:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830452.920920:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973315456:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830452.920938:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830452.920946:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830452/real 1787830452] req@ffff9c1a78a783c0 x1874668973315456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830452.920956:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973315456/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830452.920963:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830452.920966:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830452.920968:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973315456:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830452.920984:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973315584:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830452.921331:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830452.921380:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830452/real 1787830452] req@ffff9c1a78a7a1c0 x1874668973315584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830452.921393:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973315584/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830452.921401:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830452.921404:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830452.921407:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973315584:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830452.921413:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973315712:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830452.921433:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830452.921443:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830452/real 1787830452] req@ffff9c1a78a7b0c0 x1874668973315712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830452.921454:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973315712/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830452.921462:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830452.921465:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830452.921468:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973315712:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830453.431561:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830453.559464:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830453.754781:0:49200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830454.455466:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830454.583453:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:1.0:1787830454.775539:0:15103:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830455.479463:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787830455.607676:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830455.799520:0:15688:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830456.503472:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830456.633147:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:0.0:1787830456.824004:0:6181:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830457.527599:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830457.655773:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830457.847834:0:15689:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830458.039510:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4cb2f4f8-58f6-4067-a748-bb0df2a4d242->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830458.039519:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830458.039524:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830458.039537:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.148@tcp: connect to NID 192.168.203.148@tcp last attempt 6908 00000100:00080000:1.0:1787830458.039551:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.148@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830458.039561:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.148@tcp: import ffff9c1a5043d000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000100:00100000:1.0:1787830458.039603:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7b0c0 x1874668973315840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830458.039630:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830458.039633:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830458.039635:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830458.039646:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6908 00000100:00080000:1.0:1787830458.039651:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830458.039655:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a46db5000: import ffff9c1a450ef000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830458.039673:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830458.039695:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a1c0 x1874668973315968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830458.039703:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973315840:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787830458.039709:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7127393f-350f-49ff-88de-bcfd4bafe057->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830458.039714:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830458.039716:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830458.039720:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: connect to NID 192.168.203.148@tcp last attempt 6908 00000100:00080000:1.0:1787830458.039725:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830458.039729:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a46db5000: import ffff9c1a450e8800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830458.039733:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830458.039746:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a783c0 x1874668973316096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830458.039761:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830458.039765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787830458.039770:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830458.039770:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830458.039778:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6908 00000100:00080000:1.0:1787830458.039783:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830458.039788:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c1a49bbf800: import ffff9c1a4a1eb000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830458.039794:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830458.039805:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a79e00 x1874668973316224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830458.039817:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830458.039820:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787830458.039821:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830458.039824:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6908 00000100:00080000:1.0:1787830458.039827:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830458.039829:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830458/real 1787830458] req@ffff9c1a78a7b0c0 x1874668973315840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830458.039833:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c1a49bbf800: import ffff9c1a4a1ee000 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830458.039840:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830458.039855:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7b0c0 x1874668973315840/t0(0) o250->MGC192.168.203.148@tcp@192.168.203.148@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830458.039860:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7a940 x1874668973316352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830458.039878:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a5043d000 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830458.039879:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830458.039885:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830458.039888:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830458.039889:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.148@tcp failed (-110) 00000100:00080000:1.0:1787830458.039892:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6908 00000100:00080000:1.0:1787830458.039896:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830458.039899:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4cb2f4f8-58f6-4067-a748-bb0df2a4d242:2410:1874668973315840:192.168.203.148@tcp:250:kworker.0 00000100:00080000:1.0:1787830458.039903:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c1a49bbf800: import ffff9c1a4a1e9800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830458.039910:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830458.039921:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a78780 x1874668973316480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830458.039928:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973315968:192.168.203.148@tcp:38:kworker.0 00000100:00080000:1.0:1787830458.039943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 3b7082ec-d0c1-445e-b4e3-d9e48b895c0e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830458.039947:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830458.039949:0:36:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830458.039954:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: connect to NID 192.168.203.148@tcp last attempt 6908 00000800:00000100:0.0:1787830458.039958:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00080000:1.0:1787830458.039962:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830458.039968:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c1a49bbf800: import ffff9c1a4a1ea800 using connection 192.168.203.148@tcp/192.168.203.148@tcp 00000008:00100000:1.0:1787830458.039973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830458.039976:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830458/real 1787830458] req@ffff9c1a78a7a1c0 x1874668973315968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830458.039988:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c1a78a7bc00 x1874668973316608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830458.039993:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a1c0 x1874668973315968/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830458.040018:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450ef000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830458.040023:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830458.040027:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973315968:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830458.040034:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973316096:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830458.040055:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830458.040066:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830458/real 1787830458] req@ffff9c1a78a783c0 x1874668973316096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830458.040078:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a783c0 x1874668973316096/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a46db5000@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830458.040085:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a450e8800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830458.040088:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830458.040091:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a783c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7127393f-350f-49ff-88de-bcfd4bafe057:2410:1874668973316096:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830458.040096:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973316224:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830458.040115:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830458.040125:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830458/real 1787830458] req@ffff9c1a78a79e00 x1874668973316224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830458.040136:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a79e00 x1874668973316224/t0(0) o38->lustre-MDT0000-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830458.040144:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1eb000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830458.040147:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830458.040150:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a79e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973316224:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830458.040156:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973316352:192.168.203.148@tcp:38:kworker.0 00000800:00000100:0.0:1787830458.040174:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830458.040184:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830458/real 1787830458] req@ffff9c1a78a7a940 x1874668973316352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830458.040195:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7a940 x1874668973316352/t0(0) o38->lustre-MDT0001-mdc-ffff9c1a49bbf800@192.168.203.148@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830458.040202:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ee000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830458.040205:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830458.040207:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973316352:192.168.203.148@tcp:38:kworker.0 00000100:00100000:0.0:1787830458.040224:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973316480:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830458.040244:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830458.040254:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830458/real 1787830458] req@ffff9c1a78a78780 x1874668973316480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830458.040265:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a78780 x1874668973316480/t0(0) o8->lustre-OST0000-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830458.040279:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1e9800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830458.040282:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830458.040284:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a78780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973316480:192.168.203.148@tcp:8:kworker.0 00000100:00100000:0.0:1787830458.040290:0:2410:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973316608:192.168.203.148@tcp:8:kworker.0 00000800:00000100:0.0:1787830458.040308:0:2410:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.148@tcp 00000100:00100000:0.0:1787830458.040318:0:2410:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830458/real 1787830458] req@ffff9c1a78a7bc00 x1874668973316608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830458.040329:0:2410:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c1a78a7bc00 x1874668973316608/t0(0) o8->lustre-OST0001-osc-ffff9c1a49bbf800@192.168.203.148@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830458.040336:0:2410:0:(import.c:65:import_set_state_nolock()) ffff9c1a4a1ea800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830458.040339:0:2410:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.148@tcp failed (-110) 00000100:00100000:0.0:1787830458.040342:0:2410:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c1a78a7bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:3b7082ec-d0c1-445e-b4e3-d9e48b895c0e:2410:1874668973316608:192.168.203.148@tcp:8:kworker.0 00000100:00100000:1.0:1787830458.551461:0:95335:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a603ce6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787830458.679869:0:95332:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c1a43c9acc0 going to sleep for 0 seconds 00010000:00010000:3.0:1787830458.872146:0:34905:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 77493 lines, 77493 kept, 0 dropped, 0 bad.